Nov 14

How to pause and resume rsync

Just like any other Linux process, you can pause rsync by sending it a TSTP (polite) or STOP (forcible) signal. On the terminal you’ve run rsync in, pressing Ctrl+Z sends TSTP. Resume with the fg or bg command in the terminal or a CONT signal. To resume the rsync where it is interrupted, make sure to pass -P, otherwise rsync will check all files again and process the file it was interrupted on from scratch.

Ex:-

01) execute rsync command with “-P”

 

more about “-P” option from man page of rsync :

-P
The -P option is equivalent to –partial –progress. Its purpose is to make it much easier to specify these two options for a long transfer that may be interrupted.

There is also a –info=progress2 option that outputs statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify –info=name0) if you want to see how the transfer is doing without scrolling the screen with a lot of names. (You don’t need to specify the –progress option in order to use –info=progress2.)

Finally, you can get an instant progress report by sending rsync a signal of either SIGINFO or SIGVTALRM. On BSD systems, a SIGINFO is generated by typing a Ctrl+T (Linux doesn’t currently support a SIGINFO signal). When the client-side process receives one of those signals, it sets a flag to output a single progress report which is output when the current file transfer finishes (so it may take a little time if a big file is being handled when the signal arrives). A filename is output (if needed) followed by the –info=progress2 format of progress info. If you don’t know which of the 3 rsync processes is the client process, it’s OK to signal all of them (since the non-client processes ignore the signal).

CAUTION: sending SIGVTALRM to an older rsync (pre-3.2.0) will kill it.

 

02) pause rsync

Press Ctrl+Z to stop the process.

 

 

If you kill rsync with Ctrl+C it does stop it mid transfer but it will not keep any temporary files there unless running with the –partial option.

For those also interested in just pausing rsync, if you press Ctrl+Z it pauses the program. When you resume it by running fg or bg rsync will reiterate over the file that it didn’t finish and continue downloading the rest of the files. To see all currently paused programs run jobs

03) To view the jobs

 

so job id is “1”

04) resume rsync process

to resume the job, you may need to pass jobs id as fg command.

 

The general job control commands in Linux are:

 

  • jobs – list the current jobs
    fg – resume the job that’s next in the queue
    fg %[number] – resume job [number]
    bg – Push the next job in the queue into the background
    bg %[number] – Push the job [number] into the background
    kill %[number] – Kill the job numbered [number]
    kill -[signal] %[number] – Send the signal [signal] to job number [number]
    disown %[number] – disown the process(no more terminal will be owner), so command will be alive even after closing the terminal. Read the rest of this entry »

Dec 25

AWS CloudWatch Apache HTTP monitoring

AWS CloudWatch provides custom metric monitoring. It is very useful when require to monitor performance of the custom application or server. Here we are going to guide how monitor Apache HTTP server performance using AWS CloudWatch custom metrics. All the installation and configuration performed on CentOS, most of the commands work on any LINUX / UNIX like system. If you need more details, you may can visit official documentation. I always try to attach official docs where it is possible.

1) Install aws cli

 

You can find details guidelines from official documents

Once the installation is completed, you can verify installed version using following command.

2) Create IAM user with “Programmatic access” and assign following policy to the user.

 

please note down “access key ID and secret access key” which is needed on next step.

3) Configure AWS client

 

execute following command as root user. you must enter Key ID and secret key. you should enter region name where your EC2 instance is running.
please refer this link to obtain your region name code
you can keep output format as none.

4) Create simple shell script to push data into AWS Cloudwatch

 

you may can replace localhost with your EC2 instance private IP. Here we are interested to push Busy Workers,Idle worker and Connection Total data to CloudWatch, but there are few other metrics are available on server status page . you can get full list of metrics by visiting http://<your server IP>/server-status?auto

 

5) Set cron job to push data

setup cronjob to execute above shell script to run every 5 minutes

6) How to view AWS CloudWatch custom metrics

i) Go to AWS CloudWatch

ii) Then select Metrics menu from left hand side bottom.

iii) Select “All metrics” tab , and you can see “EC2:HTTP-Apache” under Custom Namespaces

iv) Example output of the graph is as follows.(you should send data frequently to CloudWatch to generate useful graph)

 

AWS CloudWatch custom metrics raph

Dec 01

Solved – AWS RDS MySQL ERROR 1227 (42000) at line : Access denied

When your are trying to import your data into RDS MySQL, it may prompt with following error message. “ERROR 1227 (42000) at line xxx: Access denied; you need (at least one of) the SUPER privilege(s) for this operation

 

Error!

ERROR 1227 (42000) at line xxx: Access denied; you need (at least one of) the SUPER privilege(s) for this operation

This can be fixed by removing the DEFINER from MySQL dump. You can use following simple command to fix this issue.

 

there are alternative solution which provided by AWS premium support knowledge base , But that does not work for me. You can try that out if I above mentioned work around does not work.

Let’s look at why there is limited permission on RDS MySQL .

As you might be aware, AWS RDS (Relational Database Service) is a managed serviceĀ  and hence in order to guarantee the stability of RDS instance, the permissions of master user (root user in RDS) are not same as root user in native mysql.

RDS Master user has the following permission:
SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER ON *.* WITH GRANT OPTION, REPLICATION SLAVE (Only For Amazon RDS MySQL versions 5.6 and 5.7, Amazon RDS MariaDB)

So if you need more permission other than above, you have to request / inform it to AWS support team. they will do the necessary arrangement.

If you have any questions please do comment below. :)

have nice time !!

Older posts «

Fetch more items