Making long/bulk web operations offline/asynchronous
As web is a stateless medium of interaction, every action is a request/response pair. As this request response pair should have a definite life cycle there are timeouts attached to It i.e. you can not wait for a request’s response forever. There comes a problem where you want to execute longish actions. These longish actions doesn’t exists in common websites as these websites restrict these actions by limiting their scopes but in some web applications these longish actions are required like CRMs where you can import contact lists of significant sizes and later manage them using CRM application. Now loading this big contact file can take longer than timeout of request/response which can problem for the user of these applications.
Another example can be taken as exporting a big report which can be done synchronously but may be problematic for user which will keep downloading it for long time.
Following are some of solutions which can be considered for above problem:
A. Queuing longish/bulk actions and notify the user later when the actions is performed for the worker thread of queue.
B. Perform action synchronously but update its status to user using different thread.
Let’s take each approach one by one.
Approach A: In this case, you need a separate infrastructure with your application which acts as queue of actions which will help you in following ways:
1. It will control the load on your application as only limited number bulky or longish actions will be allowed. Because of the queue of actions, it can be controlled and worker threads can be configured accordingly.
2. More control over reporting the status of each action. User can be notified in different ways about completion of an action started by him e.g. an email notification can be sent to him on completion of his action.
Implementation will require following:
1. A place to store this queue of actions. It can be a DB table or in-memory queue. Queue should store all the inputs required to perform that action.
2. A thread or set of threads to perform queued actions and update action’s status in queue.
3. A notification mechanism to notify the user.
Approach B: You do not require an a complete queue like infrastructure in this approach. You need to perform your long or bulk operation in following steps:
1. Create a table to store progress of long/bulk operation
2. Generate a requestID and insert a row in table created against that requestID
3. Now while doing this long/bulk operation, keep updating the progress of operation against generated requestID.
4. On web application, keep fetching the status of operation against the generate requestID and keep showing this to the user until operation is complete.
You can use Ajax in following way to display the progress of any long action using jquery:
function doLongAction(){
// Generate a requestID and store an entry against this long action
var requestId = '';
$.ajax({
url: '../portal/longAction?q=addlongOperation',
type: "POST",
cache: false,
dataType: "json",
async: false,
success: function(msg){
if(msg.result == 'success') {
requestId = msg.requestid;
} else if(msg.result == 'error') {
// Show error
}
},
error: function(msg){
// Show error
}
});
var serverUrl = '../portal/longAction?q=startLongAction&requestId=' + requestId;
$.ajax({
url: serverUrl
type: "POST",
cache: false,
dataType: "json",
async: false,
success: function(msg){
if(msg.result == 'success') {
// do nothing
} else if(msg.result == 'error') {
// show error
}
},
error: function(msg){
//show error
}
});
showlongActionProgress(requestId);
}
function showlongActionProgress(requestId) {
var progressInfo = function getProgressInfo() {
var serverurl = ''../portal/longAction?q=fetchActionDetail&requestId=' + requestId;
$.ajax({
url: serverurl,
type: "POST",
cache: false,
dataType: "json",
async: false,
success: function(msg){
if(msg.result == 'success' && msg.detail.status == 'INPROGRESS') {
// Show progress
} else if(msg.result == 'success' && msg.detail.status == 'DONE') {
// Show success
clearInterval(progressInfoHandle);
} else if(msg.result == 'error') {
// Show error
clearInterval(progressInfoHandle);
}
},
error: function(msg){
clearInterval(progressInfoHandle);
//Show Error
}
});
};
var progressInfoHandle = setInterval(progressInfo, 500);
}
Database page corruption on disk or a failed file read …. Again – Mysql
Again encountered the stopped server in the morning because of page file corruption. Reason was the bad power situation in office during night shifts and which costs us the whole day to repair the testing server.
As from the last time learnings of page file corruption (previous post) we knew how to start the server in recovery mode by changing the /etc/my.cnf file, we started it in recorvery mode 3 (innodb_force_recovery = 3) which is SRV_FORCE_NO_TRX_UNDO i.e. Do not run transaction rollbacks after recovery.
We were able to do few things which are allowed in this recovery mode like checking few databases and integrity of some of the application data. We tried to check mysql tables using mysqlcheck command.
$ mysqlcheck -u root -p --all-databases
Using above command it was not not completing the command and was giving following error
RROR 2013 (HY000): Lost connection TO MySQL server during query
Same error we got when we tried to backup all database schemas in the DB. It took backup of some of the schemas but soon gives above error. We were stuck as not able to take backup of data and also were not able to repair the tables/databases.
tried following thing and backup was successfully taken after that. Even we figured out which tables got corrupted as we could run mysqlcheck successfully also afterwards. We did following simple thing in /etc/my.cnf and restarted the server:
[mysqld] innodb_force_recovery = 4
Finally recovered all the data using the backup and server came back to life after the whole day of effort.
Google App Engine – Java Development
Loading data file into MySql Database
Loading data into mysql tables can be done using simple insert statements but to get achieve better speed and performance it requires a special tool and LOAD DATA INFILE command is provided for this purpose in MySql. Used it recently and worked really fast, remember to drop indexes before this and rebuild them in case you have bigger table to restore.
LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n';
This command is complement of SELECT … INTO OUTFILE command which can produce delimited file directly from the SELECT query and LOAD DATA INFILE can use the same file using very similar command.
LOAD DATA INFILE provides many advantages over normal INSERT statement which can be optimized upto some extent but not much. You can visit following link for getting more information about speed of INSERT statement and what optimizations can be done.
In case of LOW_PRIORITY, execution of the LOAD DATA statement is delayed until no other clients are reading from the table and If you specify CONCURRENT with a MyISAM table that satisfies the condition for concurrent inserts (that is, it contains no free blocks in the middle), other threads can retrieve data from the table while LOAD DATA is executing.
If you want to ignore foreign key constraints during the load operation, you can issue a SET foreign_key_checks = 0 statement before executing LOAD DATA.
LOAD DATA INFILE 'data.txt' FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' INTO TABLE tbl_name (col1, col2, col3...);
For more details refer here