为什么我的ansible copy模块非常慢?!

最近用ansible来做一些初始化的操作,其中有用到copy模块来拷贝一些东西,结果发现速度慢到哭,经网络监测,2G的带宽它才给我跑了10MB,百度依旧是没有好答案,只能谷歌,第一条stack overflow就给出解决方案(这里吐槽下某朝最近的严打活动。。。,没有了谷歌我们怎么活)

原文如下:

I’m using Ansible to copy a directory (900 files, 136MBytes) from one host to another:

---
- name: copy a directory
  copy: src={{some_directory}} dest={{remote_directory}}

This operation takes an incredible 17 minutes, while a simple scp -r <src> <dest> takes a mere 7 seconds.

I have tried the Accelerated mode, which according to the ansible docs “can be anywhere from 2-6x faster than SSH with ControlPersist enabled, and 10x faster than paramiko.”, but to no avail.

答案:

TLDR:

use synchronize instead of copy.

use synchronize instead of copy.

use synchronize instead of copy.

重要的事情说三遍!!!

Here’s the copy command I’m using:

- copy: src=testdata dest=/tmp/testdata/

As a guess, I assume the sync operations are slow. The files module documentation implies this too:

The “copy” module recursively copy facility does not scale to lots (>hundreds) of files. For alternative, see synchronize module, which is a wrapper around rsync.

Digging into the source shows each file is processed with SHA1. That’s implemented using hashlib.sha1. A local test implies that only takes 10 seconds for 900 files (that happen to take 400mb of space).

So, the next avenue. The copy is handled with module_utils/basic.py’s atomic_move method. I’m not sure if accelerated mode helps (it’s a mostly-deprecated feature), but I tried pipelining, putting this in a local ansible.cfg:

[ssh_connection]
pipelining=True

It didn’t appear to help; my sample took 24 minutes to run . There’s obviously a loop that checks a file, uploads it, fixes permissions, then starts on the next file. That’s a lot of commands, even if the ssh connection is left open. Reading between the lines it makes a little bit of sense- the “file transfer” can’t be done in pipelining, I think.

So, following the hint to use the synchronize command:

- synchronize: src=testdata dest=/tmp/testdata/

That took 18 seconds, even with pipeline=False. Clearly, the synchronize command is the way to go in this case.

Keep in mind synchronize uses rsync, which defaults to mod-time and file size. If you want or need checksumming, add checksum=True to the command. Even with checksumming enabled the time didn’t really change- still 15-18 seconds. I verified the checksum option was on by running ansible-playbook with -vvvv, that can be seen here:

ok: [testhost] => {"changed": false, "cmd": "rsync --delay-updates -FF --compress --checksum --archive --rsh 'ssh  -o StrictHostKeyChecking=no' --out-format='<<CHANGED>>%i %n%L' \"testdata\" \"user@testhost:/tmp/testdata/\"", "msg": "", "rc": 0, "stdout_lines": []}

E文不好的我给翻一下:

其实就是用synchronize替代copy模块,另外,在ansible.cfg文件中改如下:

[ssh_connection]
pipelining=False

然后去感受替换后的奇迹吧,骚年~

ansible 使用root用户批量修改非root用户密码

目录

一 环境

  1. 系统环境
  2. 软件版本

二  批量修改用户密码

1. hosts 文件配置

2. 修改密码的yml

正文

一 环境

  1. 系统环境
  2. 软件版本

二批量修改用户密码

  1. host 文件配置

1.1 hosts 文件

 

[sudozone]
root@192.168.2.2

1.3 changpassword.yml  –修改密码的playbook

- name: changepasswd 
  hosts: onlystablesudo 
  sudo: yes
  vars:
    change_user: username 被修改密码的用户
    passwd: $1$zYZGbfAM$7aUvB/8EGbQhKaZuBMhV90 (此处必须使用加密的密码)
  tasks:
    - name: changepasswod 
      user: name={{ change_user }} password={{ passwd }} update_password=always

1.3.1  生成加密的密码

openssl passwd  -1 “12321421421”

2. 实施

ansible-playbook changpassword.yml

Ansible之playbook自动完成硬盘分区,格式化,挂盘,并远程创建用户

摘要: 对于刚刚拿到拿到的新机器,唯一的可用信息有,IP地址,服务器登陆用户名和密码。 一般情况,root用户是不能直接登陆,只能通过普通用户跳转。 特别是最近工作中遇到的腾讯云服务器,申请之后硬盘都还没有挂载上去,还需要手动挂载操作,这个是比较费事的。所以这里采用ansible中的playbook将以上的工作都一并完成

前言

对于刚刚拿到拿到的新机器,唯一的可用信息有,IP地址,服务器登陆用户名和密码。

一般情况,root用户是不能直接登陆,只能通过普通用户跳转。

特别是最近工作中遇到的腾讯云服务器,申请之后硬盘都还没有挂载上去,还需要手动挂载操作,这个是比较费事的。所以这里采用ansible中的playbook将以上的工作都一并完成

playbook

以下的playbook功能包括:

  1. 分区新挂载硬盘,默认是一个分区全部使用
  2. 格式化新挂载分区
  3. mount新分区
  4. 创建远程管理用户
  5. 下发ssh公钥,为以后ansible远程管理做准备

- hosts: "{{ host }}"
  remote_user: "{{ loginUser }}"
  vars:
    work_path: /opt/object/server/
    disk: /dev/sdb
    partition: /dev/sdb1
    mountDir: /data
  vars_prompt:
    - name: "username"
      prompt: "Insert User Name:"
      default: "www"
      private: no
    - name: "passwd"
      prompt: "Insert Password for the user:"
      default: "123qwe"
      private: no
  tasks:
    - name: Yum Install
      yum: name="{{ item }}" state=latest
      with_items:
        - libselinux-python
      become: yes
      become_method: su

    - name: New Disk Partition
      script: ./../script/disk.sh "{{ disk }}"
      become: yes
      become_method: su

    - name: New Disk Format(ext4)
      filesystem: fstype=ext4 dev="{{ partition }}" opts="-cc"
      become: yes
      become_method: su

    - name: New Disk Mount
      mount: name="{{ mountDir }}" src="{{ partition }}" fstype=ext4 state=mounted
      become: yes
      become_method: su

    - name: Create Remote User
      user: name="{{ username }}" password="{{ passwd }}"
      become: yes
      become_method: su

    - name: Set up SSH Key
      authorized_key: user={{ username }} key="{{ lookup('file', '/home/eric/.ssh/id_rsa.pub') }}"
      become: yes
      become_method: su

自动化分区脚本(disk.sh)

#!/bin/bash

DISK=$1

CHECK_EXIST=`/sbin/fdisk -l 2> /dev/null | grep -o "$DISK"`
[ ! "$CHECK_EXIST" ] && { echo "Error: Disk is not found !"; exit 1;}

echo "1" > /tmp/disk.log

CHECK_DISK_EXIST=`/sbin/fdisk -l 2> /dev/null | grep -o "$DISK[1-9]"`
[ ! "$CHECK_DISK_EXIST" ] || { echo "WARNING: ${CHECK_DISK_EXIST} is Partition already !"; exit 1;}

echo "2" > /tmp/disk.log

/sbin/fdisk /dev/sdb<<EOF  
d
n
p
1
1

t
83
w
EOF

执行指令

ansible-playbook server_init.yml -vvv -k --ask-become-pass -e "loginUser=eric" -e "host=192.168.1.101"

ansible常用的一些方法总结

1.把ansible这台机器上的/etc/hosts远程scp到webserver组下面的所有机器上/root/hosts
ansible webserver -m copy -a “src=/etc/hosts dest=/root/hosts”
2.修改webserver组下面所有的机器上的文件/root/hosts的属主:test,权限755
ansible webserver -m file -a “dest=/root/hosts mode=755 owner=test group=test”

3.在webserver组下面所有的机器上创建kk目录,属主:test,权限755
ansible webserver -m file -a “dest=/root/kk mode=755 owner=test group=test state=directory”

4.删除目录kk(递归的删除)和删除文件
ansible webserver -m file -a “dest=/root/kk state=absent”

5.采用shell的模块在root下创建kk文件,-f 10(一般在配置文件里面写好)是以并行的方式去执行,会fork出10个子进程,速度变快
ansible webserver -m shell -a ‘touch /root/kk’ -f 10

6.指定以test用户,该test用户是指webserver下所有机器的test用户,以普通用户test执行命令
ansible webserver -a “/usr/bin/foo” -u test –ask-sudo-pass

7.确认软件包wget已经安装,确认wget软件包还没有安装
ansible webserver -m yum -a “name=wget state=present”
ansible webserver -m yum -a “name=wget state=absent”

8.确认某个服务在所有的webservers上都已经启动,重启,确认服务已经停止
ansible webserver -m service -a “name=httpd state=started”
ansible webserver -m service -a “name=httpd state=restarted”
ansible webserver -m service -a “name=httpd state=stopped”

9.如果运行命令后,不想获取返回的信息
ansible all -B 3600 -P 0 -a “/usr/bin/long_running_operation –do-stuff”
如果你确定要在命令运行后检查运行的状态,可以使用 async_status 模块.前面执行后台命令后会返回一个 job id, 将这个 id 传给 async_status 模块
ansible web1.example.com -m async_status -a “jid=488359678239.2844”
获取状态的命令,其中 -B 1800 表示最多运行30分钟, -P 60 表示每隔60秒获取一次状态信息
ansible all -B 1800 -P 60 -a “/usr/bin/long_running_operation –do-stuff”

10.在执行一个 playbook 之前,想看看这个 playbook 的执行会影响到哪些 hosts
ansible-playbook playbook.yml –list-hosts

11.ansible-playbook执行相应的rabbitmq_pp_restart.yml脚本
sshpass -p $PP_Pass ansible-playbook rabbitmq_pp_restart.yml –ask-pass -i /root/ansible-playbook/ycq/ycq.host

12.ansible的script模块的执行方式,主要把本地的脚本推送到远端,在远端执行该脚本
sshpass -p $Prd_Pass ansible check-port-8080 -m script -a “/home/appuser/zabbix_ycq/scripts/check_tomcat_port.sh” -i /home/appuser/zabbix_ycq/ansible-playbook/ycq.host –ask-pass