Friday, April 5, 2013

Restoring MySQL Databases


May 21st, 2008 By: brian

This is a quick post to possibly save someone some time when they google for restoring mysql data files. The gist of it is, don’t try to copy InnoDB files directly into your data directory, it won’t work.
Background
Sometime back I wiped my development box clean and reinstalled the OS from scratch. It’s a wonderful feeling to start fresh and get rid of all the old whacky configs and random junk that collects on a filesystem. Prior to this cleansing, I backed up all the junk I considered save-worthy. One thing I saved was they MySQL data directory. Many of the databases were old projects and at the time I didn’t think I’d need to look at them anytime soon. Well, anytime soon arrived yesterday in the form of me really wanting to see how I had implemented a particular feature in an old project. The database component of that project was essential, so I couldn’t just glance at the code, I wanted to get it running again with the database backend.
Problem
I probably should have known better, but I decided to just try to shutdown mysql and copy the files directly to my new data directory. After some permission and user adjusting, that seemed to work… until I actually tried to access the restored database I needed, which used mostly InnoDB tables. They were not showing up or working properly. A quick glance revealed a file I hadn’t copied which seemed like it *might* be something important:
$ ls -lh /Backup/mysql/data/
-rw-rw----    1 brian  staff   1.0G May 21 10:05 ibdata1
Yeah, oops. So InnoDB stores things a bit differently than MyISAM.
Solution
The solution turns out to be simple and is what I should have done in the first place.
First, shutdown the mysql server:
$ mysqladmin -u root -p shutdown
Next, startup mysqld with the –datadir option pointing to your old data directory:
$ mysqld --datadir=/Backup/mysql/data/
It may complain about some issue with writing to log files, but that doesn’t matter for this quick export.
Now just dump your database as you normally would (or should have originally):
$ mysqldump -u root -p database_name > database_name.sql
Lastly, shutdown mysqld again, start mysqld normally and then do the import as usual:
$ mysql -u root -p database_name < database_name.sql
(you’ll likely need to create the database first)
There you have it – the way to get your old database files restored into your current setup. There is probably some better easier way to do this, and if someone knows it, clue me in will ya?

2 Responses to “Restoring MySQL Databases”

  1. Cassidy Says:
    You should have been able to tar up the whole data directory and extract it in its new home. Assuming the paths were the same between installs, (if not, edit your my.cnf) innodb should happily come up just fine without missing a beat. Of course you’d want to have done the tarring with MySQL cleanly shutdown, otherwise you’d probably have some issue with ibdata files and probably the MyISAM files too.
    Another gotcha is if you change architectures and try and copy the ibdata files to a new box.
  2. brian Says:
    Cassidy – ah, thanks for the tip. I still probably needed to use the method I did since I wasn’t moving an entire data directory to an empty new one. I was only moving part of the old into my “new” (been using it for 8 months) dev environment.

Recover a deleted file in Subversion


May 29th, 2008 By: brian

SVN is great for keeping our projects under tight control. Occasionally, we have the need to get something back that was deleted in a previous revision. So I can remember how to do it next time, here it is:
$ svn copy -r 1234 url/to/deleted/file path/to/recovered/file
This will copy the file at the revision specified to the new file in the “restore to path” part. You can find the revision by doing an ’svn log –verbose’ of the directory it was in. That’s all there is to it!
My theme seems to be restoring and recovering… is that a bad thing?

Fragment caching with Radiant CMS


June 10th, 2008 By: Wes Bangerter

We’re in the process of converting our website to Radiant CMS, and one of the new things on the site is a “Blog Blurbs” section at the bottom of every page that lists our latest blog post. Our blog is in Wordpress, so I setup the RSS Reader extension in Radiant to fetch the posts. Everything worked great, except that page loading was noticeably slower. After the page gets cached everything is fine but this is included on every page so going through the site when there were not cached pages was really frustrating. I thought about modifying the RSS Reader extension to cache our blog blurbs, but I figured a more general approach would work better, so I came up with a fragment caching extension. We’re using it like this:
<r:cache name="rss_fragment" time="60">
  <r:feed:items url="[feed_url]" limit="1">
    <h3><r:feed:link /></h3>
    <p><r:feed:content max_length="300" no_html="true" /></p>
  </r:feed:items>
</r:cache>
The code is at github.

Ruby Timing Shortcut


June 10th, 2008 By: Daniel

Recently I was trying to optimize my code a little and needed a quick way to compare the speed of different code snippets. Of course Rails comes with the Benchmark module for doing just that. It is, however, in my opinion a bit clunky for quick tests. Look at all the typing it takes just to find the average speed of a snippet over 100 iterations:
Benchmark.bm do |x|
  x.report do
    100.times do
      (1...1000).to_a.sum
    end
  end
end
So I tossed this into my ~/.irbrc file:
class Integer
  def ti(&block)
    Benchmark.bm do |x|
      x.report do
        self.times do
          yield
        end
      end
    end
  end
end
And now I can “time it” like this:
>> 100.ti {(1...1000).to_a.sum}
      user     system      total        real
  0.540000   0.050000   0.590000 (  0.597664)
=> true
Much nicer!

How We’re Using VMWare Server


July 15th, 2008 By: mark

VMWare releasing a free version of their server virtualization product opened up some possibilities for us. Moki Systems had multiple servers performing a variety of tasks but there were things we didn’t have. We didn’t have a split between production and development servers and we didn’t have good utilization. Low CPU and memory utilization offers a comfort level but it adds costs, especially when you need more servers that cost money, use electricity, and generate heat.
Free VMWare server limits running virtual servers to 4. Our solution was to purchase new servers equipped with 1 SATA disk for the OS and 4 SCSI disks, 1 for each virtual machine (VM). While VMWare offers an option to write to the disk directly I opted not to do that because the virtual disk files VMWare uses make your VM’s highly portable and easy to duplicate. Disaster recovery practice becomes far easier when you can easily make a duplicate of your production server.
This architecture’s utility was brought to light when one of your servers died. We were able to pull the disks containing our production servers, copy the VM’s to another server, then get those servers running again quickly. There was no need for OS reinstall or recovery from backup and because of this portability downtime was minimized.
I have VMWare server installed on my Windows workstation with a Centos 5 VM running in the background. I use it for some MySQL and PHP stuff so it isn’t used a whole lot but I hardly know it’s there. The beauty of this that the Windows guys can have a Linux VM and the Linux guys can have a Windows VM. That can help quite a bit when testing browser compatibility on a website or getting access to a tool not available for your OS of choice.
VMWare server has helped us utilize our server resources better at a very minimal cost. Running 4 servers for the price of one saves us time and money. What else could you ask for?

Ajax multi file upload with php, iframe, & javascript


July 22nd, 2008 By: andrew

I found several solutions after browsing around google trying to find a multi-file uploader built around php. Most of these solutions for multi-file uploading fit into one of these three categories.
  1. Use Flash to upload the images. (The browser only supports one image at a time) –SWFUpload
  2. Use Javascript to add multiple file type fields, while hidding previous fields – the-stickman
  3. Use Javascript and hidden iframes embedded on the same page to upload the images. –ajaxf1
I decided to take pieces of the mentioned methods and come up with a solution that worked for me.
Here is the idea behind what we are going to be doing.
  1. Create a HTML form which has a file upload field
  2. Set the target of this HTML form to an iFrame which is on the same page
  3. Have Javascript submit the form everytime the “file” field changes
  4. Javascript hides the form when submitted, and displays an “animation”
  5. PHP takes the uploaded file, thumbnails it, and stores the image data
  6. Javascript unhides the form, and re-hides the animation when upload is complete
  7. Javascript and php show the thumbnail on the same page right after it was uploaded
This makes this little script appear like an ajax application. It is NOT actually AJAX, but uses css, iframes, php sessions, and javascript to make it appear so.
I’ll try to explain what is happening at each step, but keep in mind, I’m not an expert programmer like my fellow co-workers, and I borrowed most of the work from the mentioned sources above. If you just want to download all the files and take the easy way out, skip to the bottom of the post.
Steps 1 & 2 & 3 – Make Our Form
1
2
3
4
5
6
7
8
9
<form name="pictureform" action="upload.php" method="post" enctype="multipart/form-data" target="upload_target" >
     <p id="f1_upload_process">Loading...<br/><img src="loader.gif" /><br/></p>
     <p id="f1_upload_form" align="center"><br/>
         <label>Add Pictures:  
              <input name="myfile" type="file" size="30" onchange="startUpload();document.pictureform.submit();" />
         </label>      
     </p>
     <iframe id="upload_target" name="upload_target" src="#" style="width:0;height:0;border:0px solid #fff;"></iframe>
</form>

Explanation of the mentioned code:
enctype="multipart/form-data"
Tells the browser you are submitting files, or images with your form.
onchange="startUpload();document.pictureform.submit();
The onchange event of the input field calls the function startUpload(), which will hide the form, and show the animation while uploading. The document.pictureform.submit(); Submits the form to our “action” which we have as “upload.php”
Step 4 – Make the JavaScript functions for hiding and showing
1
2
3
4
5
function startUpload(){
      document.getElementById('f1_upload_process').style.visibility = 'visible';
      document.getElementById('f1_upload_form').style.visibility = 'hidden';
      return true;
}
Explanation of the mentioned code:
By giving our HTML elements id tags,such as id=f1_upload_process
we make them available for manipulation by JavaScript via the DOM.
document.getElementById('f1_upload_process').style.visibility = 'visible';
says: “get the element in the document that has an id of “f1_upload_process” and make it visible.”
This then turns on the visibility of our “animation”
block which we have set “invisible” by default in our CSS file. The opposite occurs for the form block, we make it “invisible” or “hidden.”
Step 5 – Create our “upload.php” script to handle thumbnail and copy image to filesystem
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
<?php
 // keep our session
 session_start();
 
 // Check the upload file type
 
 $filetype = $_FILES['myfile']['type'];
 $gd_function_suffix = array(  
   'image/pjpeg' => 'JPEG', 
   'image/jpeg' => 'JPEG', 
   'image/gif' => 'GIF', 
   'image/bmp' => 'WBMP',
  'image/png' => 'PNG',
   'image/x-png' => 'PNG' 
  );
  $function_suffix = $gd_function_suffix[$filetype];
 
  // Build Function name for ImageCreateFromSUFFIX 
  if(!$function_suffix){
   $err = "Unsupported file type, must be JPG|PNG|GIF|BMP";
  }
  else{
  $function_to_read = 'ImageCreateFrom' . $function_suffix;
 
 // Get the image and create a thumbnail
 $img = $function_to_read($_FILES['myfile']['tmp_name']);
 if (!$img) {
  $err =  "could not create image handle";
 }
 
 $width = imageSX($img);  // Get the image width
 $height = imageSY($img); // Get the image height
 
 if (!$width || !$height) {
  $err =  "No Width or Height";
 }
 
 // Build the thumbnail
 $target_width = 100;  // Thumnail width
 $target_height = 100; // Thumbnail height
 $target_ratio = $target_width / $target_height;  //New target ratio
 
 $img_ratio = $width / $height;  // Original ratio
 
 if ($target_ratio > $img_ratio) {
  $new_height = $target_height;
  $new_width = $img_ratio * $target_height; // Tall Image
 } else {
  $new_height = $target_width / $img_ratio; // Wide Image
  $new_width = $target_width;
 }
 
 if ($new_height > $target_height) {
  $new_height = $target_height;
 }
 if ($new_width > $target_width) {
  $new_height = $target_width;
 }
 
 $new_img = ImageCreateTrueColor($target_width, $target_height); // Create a new image handle for thumbnail
 $white = imagecolorallocate($new_img, 255, 255, 255); // Allocate the color white
 if (!@imagefilledrectangle($new_img, 0, 0, $target_width-1, $target_height-1, $white)) { // Fill the image white
  $err = "Couldn't fill image";
 }
 // Copy the uploaded image to the newly created image
 if (!@imagecopyresampled($new_img, $img, ($target_width-$new_width)/2, ($target_height-$new_height)/2, 0, 0, $new_width, $new_height, $width, $height)) {
  $err = "couldn't copy new image";
 }
 
 // Create php Session array for image information
 if (!isset($_SESSION["file_info"])) {
  $_SESSION["file_info"] = array();
 }
 
 // Use a output buffering to load the image into a variable
 ob_start();
 imagejpeg($new_img);
 $imagevariable = ob_get_contents();
 ob_end_clean();
 
 // Create a unique id or name for the image
 $file_id = md5($_FILES["myfile"]["tmp_name"] + rand()*100000);
 $new_name = $file_id.".$function_suffix";
 $temp_name = $_FILES["myfile"]["tmp_name"];
 
 // assign upload directory, and target path for image
 $uploaddir = getcwd(); //
 $target_path = $uploaddir.DIRECTORY_SEPARATOR.$new_name;
 
  // Move uploaded image to uploaddir location
 if(!@move_uploaded_file($temp_name, $target_path)){
  $err = "Couldn't Copy File to Filesystem";
 }
 // Create a Session array with name, type, and image object for use in displaying thumbnails
 if (!isset($_SESSION["file_info"][$file_id])) {
  $_SESSION["file_info"][$file_id] = array(          "name" => $new_name,         "type" => $function_suffix, 
      "obj" => $imagevariable        );
 }
}
   // If no error, then pass good result and file id to Javascript for generating the thumbnail.
   // Pass our $err varialbe along
 
 if(!$err){
  $result = "'1','".$file_id."'"; 
 }
 else{
     $result = "'.$err.','".$file_id."'";
    }
 
   sleep(1); // in case of really fast upload, sleep script to show loading animation for a bit longer
 
 // Javascript calls the "stopUpload" function with its paramaters
?>
 
<script language="javascript" type="text/javascript">window.top.window.stopUpload(<?php echo $result; ?>);</script>
Explanation of Code:
These php functions make use of the GD library. You must have that enabled in PHP in order to use this. Also make sure your current directory is writeable.
I won’t explain to many things here, as the code has inline comments. You could make this code a function and have it thumbnail and save multiple sizes to the filesytem. Currently it simply thumbnails a 100 X 100 and copies the original to the current directory the script is running from. It will do GIF, JPEG, BMP, or PNG. Error checking and Session handling could be improved.
Step 6 – Create JavaScript function to show form, and hide animation when upload is finished
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
function stopUpload(success,fileid){
      var result = '';
      if (success == 1){
         result = '<span class="msg">The file was uploaded successfully!<\/span><br/><br/>';
   addImage("thumbnail.php?id=" + fileid);
      }
      else {
         result = '<span class="emsg">There was an error during file upload!' + success + '<\/span><br/><br/>'; 
      }
      document.getElementById('f1_upload_process').style.visibility = 'hidden';
      document.getElementById('f1_upload_form').innerHTML = result + '<label>File: <input name="myfile" type="file" size="30"  onchange="startUpload();document.pictureform.submit();" /><\/label>';
      document.getElementById('f1_upload_form').style.visibility = 'visible';      
 
   return true;   
 
}
Explanation of Code:
From our upload.php script, we passed two varialbes to this JavaScript function, Success and File Id.
If we had a successful file upload, we throw up a message saying “success” and call another function called addImage("thumbnail.php?id=" + fileid); <- More on this later...
If we didn't have a successful upload, we return a message saying so.
The last few lines of the function hide the "animation" bar, and recreate our form input element so we can upload another image, and do the whole thing over again. All without leaving the page!
Step 7 – Our Thumbnail.php script and our addImage() Javascript function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<?php
 // This script accepts an ID and looks in the user's session for stored thumbnail data.
 // It then streams the data to the browser as an image 
session_start();
 
$image_id = isset($_GET["id"]) ? $_GET["id"] : false; // get the id from the URL
if ($image_id === false) {
  exit(0);
 }
 if (!is_array($_SESSION["file_info"]) || !isset($_SESSION["file_info"][$image_id])) {
 
 }
 header("Content-type: image/jpeg") ;
 header("Content-Length: ".strlen($_SESSION["file_info"][$image_id]));
 echo $_SESSION["file_info"][$image_id][obj];
 exit(0);
?>
Code Explanation:
This bit of code simply gets the image id from the URL and pulls the image thumbnail data we saved into our session in our upload.php script, and outputs the image to the browser.
addImage() Javascript Function
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
function addImage(src) {
 var newImg = document.createElement("img");
 newImg.style.margin = "5px";
 
 document.getElementById("thumbnails").appendChild(newImg);
 if (newImg.filters) {
  try {
   newImg.filters.item("DXImageTransform.Microsoft.Alpha").opacity = 0;
  } catch (e) {
   // If it is not set initially, the browser will throw an error.  This will set it if it is not set yet.
   newImg.style.filter = 'progid:DXImageTransform.Microsoft.Alpha(opacity=' + 0 + ')';
  }
 } else {
  newImg.style.opacity = 0;
 }
 
 newImg.onload = function () {
  fadeIn(newImg, 0);
 };
 newImg.src = src;
}
 
function fadeIn(element, opacity) {
 var reduceOpacityBy = 5;
 var rate = 30; // 15 fps
 
 
 if (opacity < 100) {
  opacity += reduceOpacityBy;
  if (opacity > 100) {
   opacity = 100;
  }
 
  if (element.filters) {
   try {
    element.filters.item("DXImageTransform.Microsoft.Alpha").opacity = opacity;
   } catch (e) {
    // If it is not set initially, the browser will throw an error.  This will set it if it is not set yet.
    element.style.filter = 'progid:DXImageTransform.Microsoft.Alpha(opacity=' + opacity + ')';
   }
  } else {
   element.style.opacity = opacity / 100;
  }
 }
 
 if (opacity < 100) {
  setTimeout(function () {
   fadeIn(element, opacity);
  }, rate);
 }
}
Code Explanation:
I borrowed most of this code from the SWFUpload project. The basic idea is that we create a new image tag via javascript, give it a src of “thumbnail.php?id=XXXXX” XXXXX being our image name, and call a “fadeIn” function which does just that, fade the image in for a lovely effect. I have this javascript included in my index.php page via
<script type="text/javascript" src="handlers.js"></script>
Thats it! Hope you learned as much as I did from this tutorial.
Important Notes:
-This script does not check file size, or limit the number of files that can be uploaded.
-You will need to add in appropriate error checking on the uploaded files for security.
-You will want to clean out the data from the session when done with the uploading.
-Make sure you have write permission the directory where images are being saved
-You must have GD library installed for this script to work
All the files you need to be up and running are found here -> ajax-file-upload-mokisystems

16 Responses to “Ajax multi file upload with php, iframe, & javascript”

  1. Sean Says:
    Hello. I am running into a problem using your little script here for multi-file uploading. In my xampp environment on my laptop I am running php version 5.2.5 and the code works completely fine in both Firefox 3.0 and IE7. However when I upload it to my website, any image file I upload will upload just fine, yet the thumbnail that is supposed to be generated does not display.
    If I view the source I can see where it creates an img tag like but it does not display that image. Any ideas on why this might be?
    Since the code is the same from my laptop to my website and the only thing different is the version of php that is running between the two the only thing I can think of is that there is something missing from the version of php that my website is running which is php version 4.4.8. Any help would be greatly appreciated.
    Thanks,
    Sean Meyer
  2. andrew Says:
    Sean
    It could be a number of things. It sounds like the “thumbnail.php” is not outputting it to the browser. It is also possible that the upload.php file is adding the thumbnail picture id to the Session. Try commenting out the following line of code in the thumbnail.php:
    Make
    header("Content-Length: ".strlen($_SESSION["file_info"][$image_id]));
    Becomes
    //header("Content-Length: ".strlen($_SESSION["file_info"][$image_id]));
  3. Sean Says:
    Hey thanks, that worked perfectly!. I appreciate the quick response.
    Sean
  4. Ken Says:
    This is a great script, i just need some help on how to actually create and save a thumbnail of the image being uploaded, like “imageid_th.jpg”
  5. Ken Says:
    ^^ I spoke too soon!
    After you create the unique id or name for the image, create one for the thumbnail as well:
    ####//// create a thumbnail image as well
    $thumb_name = $file_id.”_th.$function_suffix”;
    imagejpeg($new_img, $thumb_name);
    And then you can upload it just like you did the image:
    ###// Move thumbnail image to uploaddir location
    if(!@move_uploaded_file($thumb_name, $target_path)){
    $err = “Couldn’t Copy Thumbnail to Filesystem”;
    }
    Thanks again for this great script!
  6. Sven Says:
    was looking for such a detailed tutorial for quite a while. very cool. however i’m having problems with FF3. in other browsers it’s working fine but with FF3 all i get is “There was an error during file upload!.Unsupported file type, must be JPG|PNG|GIF|BMP.”. of course i’m trying to upload a jpg. any ideas?
  7. apex Says:
    nice script but thumb’s are not showing. I have tryed suggestions above, but nothing work’s. When i go trough firebug i don’t have any errors in code, but something is not working. Im stuck here please suggest? I use php5 and GD is working fine, tryed in firefox3 and ie6
  8. apex Says:
    here, i setup a simple demo (no files are changed).
    try to upload more than 2 images with no error. i can’t
  9. push Says:
    any idea how to display thumbnails using ajax…. i create the thumbnails on the fly in a php but unable to show that ouput using ajax to the browser… although i f i don use ajax i can see that on browser
  10. angelivene Says:
    hi, would like to know how to upload multiple files? i already tried the script but unable to select multiple files
  11. Jamie Says:
    How do I change the script so that it doesnt change the name of the file so if I upload googleLogo.jpg I want the file name on the server to be googleLogo.jpg
  12. Srini Says:
    Hi,
    The demo script is working great!
    Can you please help me on how to retrict to upload only 4 or 5 etc., files.
  13. Jason Says:
    Demo is working great! What is about remove and session destroy? When I don’t want picture then I decided to remove picture but still session keep there should remove session, too.
    For example:
    I click upload file for picture then uploaded then made a mistake I should click remove then remove picture and memory session should remove like destroy session.
    That’s idea.
  14. CTroy Says:
    Hi,
    Love the demo script it is working great…
    Hope you do more of these, this was fun… :)
  15. Justin - Web Design Dublin Says:
    Thanks for the script. I wrote my own image handler with codeigniter but you JavaScript came in very handy. Thanks.
  16. Andy Says:
    I tried Ken’s code and work fine, but can’t save thumbnail image to another folder. Each time the thumbnail image is saved in same folder near original image