Two Spring MVC tutorial projects - Spring 4, Eclipse, Tomcat, multiple models, MultipartFile

Today I present two "hello world"-type Java EE projects using Spring MVC. I've been getting up to speed with Spring for a while now. I've taken advantage of several helpful tutorials, but I found all of them lacking in one way or another.

First, although there are quite a few Spring MVC "hello world" tutorials, I couldn't find one that accommodated all of my requirements, so I created the springmvcshell project. It's based on mkyong's excellent Spring 3 MVC Hello World Example, which I modified to:

• Use the Maven standard directory structure
• Work with the Eclipse IDE
• Work with Tomcat
• Use Spring 4 rather than an earlier version
• Use Java 1.7 rather than an earlier version

Second, I wanted to create an MVC view and corresponding controller method that include fields from two models plus a multipart file upload in a single form. Again, I found several relevant tutorials, but none that met all my needs. So I created the springmvcgoodness project, which provides all the features of springmvcshell, plus:

• Uses a single form to read and write fields from two separate classes
• Includes a file upload control using Spring's MultipartFile interface

The README files in the springmvcshell and springmvcgoodness repos explain how to install and run the two projects. You can of course see all the source code in those two repos. The remainder of this post discusses a few interesting points about the code and settings.

First of all, the Maven pom.xml files for the springmvcshell and springmvcgoodness projects are of critical importance. Both pom.xml's include the dependencies for Spring 4.0.0, as well as a reference to the com.springsource.repository.maven.snapshot repository that's required for Maven to find these dependencies. The springmvcgoodness pom.xml also adds dependencies for  the Java Server Pages Standard Tag Library (JSTL), the Java Persistence API, and Apache Commons FileUpload.

Next, the project settings are critical to making these projects work with Eclipse and Tomcat. If you clone my springmvcshell and springmvcgoodness repos, you'll automatically have the correct settings, which are stored in the .classpath and .project files. However, it's worth understanding some of the details.

• I converted these projects to faceted form (right-click the project and select Configure | Convert to Faceted Form) and added the Dynamic Web Module facet version 3.0. This makes the projects usable as servlets.
• The Java Build Path is key to avoiding compile-time errors due to missing dependencies. I had to add some libraries to the build path (right-click the project and select Build Path | Configure Build Path, then click the Add Library button). I had to add the EAR Libraries. To eliminate some compiler warnings, I also removed the old version 1.5 JRE System Library and replaced it with the JRE System Library [jre7].
• The Deployment Assembly settings are key to successful deploying on Tomcat and are accessed by right-clicking the project, selecting Properties, and then clicking Deployment Assembly. It's critical that the Maven Dependencies be added here if not already present.

Finally, here are a few points about the springmvcgoodness project:

• The approach I took to including fields from two models in one form was to create a wrapper class containing an instance of each model class.
• Once you have such a wrapper class, this view shows how to set the commandName and path attributes to refer to fields of the two model classes.
• Keys to making the file upload work include:
• Adding an HTML input tag with type="file" in the view.
• Adding a CommonsMultipartResolver bean to the servlet configuration XML file.
• Including a parameter like @RequestParam("myfile") MultipartFile myFile in the controller method.

I hope you'll find these example Spring MVC projects helpful!

Installing Selenium WebDriver and Python on Amazon Linux

I'm configuring a headless server, hosted on AWS, for browser-based automated testing. I'll use Selenium WebDriver for browser automation, and I've chosen Python as the programming language.

I've seen instructions for setting up Selenium headless automated testing in Ubuntu. However, I decided to try Amazon Linux instead of Ubuntu, so those instructions were a good start, but not exactly what I needed.

I also borrowed heavily from this excellent article on installing Firefox on Amazon Linux.

And my Python code is straight from the example in the Selenium documentation.

Here are the steps that worked for me:

One-time initial setup:

Creating an AWS Instance:

• Log in to AWS and create a new Instance from Amazon Linux AMI 2013.09.1 (64-bit).
• I made my Instance a t1.micro, but you might want to choose something more powerful that will run faster.
• I created a new Security Group. You can do the same, or use an existing one. At a minimum, you must give your IP address access on port 22 (SSH).
• I associated my existing keypair with the Instance. If you don't already have a keypair, you can create a new one.
• I named my Instance tester. You can of course name it anything you like, or even leave the name blank.

Connect to the Instance using your favorite SSH client. I used the default browser-based Java client. The username is ec2-user.

Install Selenium (which requires pip, which in turn requires setuptools):

wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
sudo python ez_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
sudo python get-pip.py
sudo pip install selenium

Install Firefox (which requires the Gimp Tool Kit):

• Use your favorite text editor (I used vim) to create a new file named gtf-firefox.sh, and insert these lines:

#!/bin/bash
# GTK+ and Firefox for Amazon Linux
# Written by Joseph Lawson 2012-06-03
# http://joekiller.com
# http://joekiller.com/2012/06/03/install-firefox-on-amazon-linux-x86_64-compiling-gtk/
 
# chmod 755 ./gtk-firefox.sh
# sudo ./gtk-firefox.sh
 
 
TARGET=/usr/local
 
function init()
{
export installroot=$TARGET/src
export workpath=$TARGET
 
yum --assumeyes install make libjpeg-devel libpng-devel \
libtiff-devel gcc libffi-devel gettext-devel libmpc-devel \
libstdc++46-devel xauth gcc-c++ libtool libX11-devel \
libXext-devel libXinerama-devel libXi-devel libxml2-devel \
libXrender-devel libXrandr-devel libXt dbus-glib
mkdir -p $workpath
mkdir -p $installroot
cd $installroot
PKG_CONFIG_PATH="$workpath/lib/pkgconfig"
PATH=$workpath/bin:$PATH
export PKG_CONFIG_PATH PATH
 
bash -c "
cat << EOF > /etc/ld.so.conf.d/firefox.conf
$workpath/lib
$workpath/firefox
EOF
ldconfig
"
}
 
function finish()
{
    cd $workpath
    wget -r --no-parent --reject "index.html*" -nH --cut-dirs=7 http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/linux-x86_64/en-US/
    tar xvf firefox*
    cd bin
    ln -s ../firefox/firefox
    ldconfig
}
 
function install()
{
    wget $1
    FILE=`basename $1`
    if [ ${FILE: -3} == ".xz" ]
       then tar xvfJ $FILE
       else tar xvf $FILE
    fi
SHORT=${FILE:0:4}*
    cd $SHORT
    ./configure --prefix=$workpath
    make
    make install
    ldconfig
    cd ..
}
 
init
install ftp://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.xz
install http://download.savannah.gnu.org/releases/freetype/freetype-2.4.9.tar.gz
install http://www.freedesktop.org/software/fontconfig/release/fontconfig-2.9.0.tar.gz
install http://ftp.gnome.org/pub/gnome/sources/glib/2.32/glib-2.32.3.tar.xz
install http://cairographics.org/releases/pixman-0.26.0.tar.gz
install http://cairographics.org/releases/cairo-1.12.2.tar.xz
install http://ftp.gnome.org/pub/gnome/sources/pango/1.30/pango-1.30.0.tar.xz
install http://ftp.gnome.org/pub/gnome/sources/atk/2.4/atk-2.4.0.tar.xz
install http://ftp.gnome.org/pub/GNOME/sources/gdk-pixbuf/2.26/gdk-pixbuf-2.26.1.tar.xz
install http://ftp.gnome.org/pub/gnome/sources/gtk+/2.24/gtk+-2.24.10.tar.xz
finish
 
 
# adds the /usr/local/bin to your path by updating your .bashrc file.
cat << EOF >> ~/.bashrc
PATH=/usr/local/bin:\$PATH
export PATH
EOF

• chmod 755 ./gtk-firefox.sh
• sudo ./gtk-firefox.sh
• On my t1.micro Instance, this step took over an hour to run!

Create a "hello world" Selenium automation program in Python:

• Use a text editor to create a file named tester.py, and insert these lines:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# go to the google home page
driver.get("http://www.google.com")

# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")

# type in the search
inputElement.send_keys("cheese!")

# submit the form (although google automatically searches now without submitting)
inputElement.submit()

# the page is ajaxy so the title is originally this:
print driver.title

try:
    # we have to wait for the page to refresh, the last thing that seems to be updated is the title
    WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))

    # You should see "cheese! - Google Search"
    print driver.title

finally:
    driver.quit()

Set up the ability to run Firefox headless:

• Append this line to .bashrc: export DISPLAY=:10
• sudo yum install Xvfb

Step that must be performed after each time you reboot the Instance:

sudo Xvfb :10 -ac

Steps that must be performed each time you connect to the Instance after disconnecting:

• Connect to the Instance using an SSH client.
• Now open a second SSH session.
• If you try to run Firefox in the first SSH session, you’ll get an error, Xlib:  extension "RANDR" missing on display ":10".)
• firefox
• python tester.py

If it's working, you'll see the output Google, followed by cheese! - Google Search.

A Python program to FTP a mix of text and binary files

When using FTP to download files from a server to your local computer, you must of course be sensitive to line endings. My server is a Linux box, so line endings are just LF. My laptop runs Windows 7, with CRLF line endings.

I recently wrote a Python program to automate the backup of files from the server to my laptop, and had to take this difference into account.

To make matters worse, some of the text files on the server have CRLF, even though it's a Linux box. The files came from a variety of sources -- uploaded by different developers, created in WordPress admin, etc -- and therefore don't have consistent line endings.

And there's another wrinkle: Some of the files are UTF-8 encoded and contain characters that can't be represented as ASCII.

At first, I thought I would simply use the Python ftplib module's retrbinary() method for binary files and retrlines() for text files. That didn't work out. The biggest obstacle was errors that resulted with the UTF-8 files when trying to read a line of text that contained a non-ASCII character and write it to a file. Two lesser hurdles were (1) figuring out when it was really necessary to add a CR, vs when the CR was already present, and (2) the possibility that the last line of the file doesn't end with a newline of any sort.

Eventually, I hit on a simple approach that almost (but not quite) 100% effective. It's good enough for my requirement, which is just to create a backup that's usable in case of disaster. My program uses retrbinary() for ALL files. If the file extension indicates a binary file, no additional action is taken. If the file extension indicates a text file, we replace every occurrence of the byte 0x0a (LF) with the two bytes 0x0d and 0x0a (CRLF) -- unless the 0x0a was already preceded by 0x0d, in which case we leave it alone.

Why is this only ALMOST 100% effective? Because it doesn't handle the case where a LF is the first byte in a buffer read by retrbinary(), and was preceded by a CR in the PREVIOUS buffer. In this case, my program inserts an extraneous CR. It wouldn't be terribly hard to fix, but not worth the trouble for my purpose.

Here's a simplified excerpt of the Python code:

from ftplib import FTP

# This is the callback function for retrbinary().
def processBytes(buffer):
 # Use the file that was opened in retrieve()
 global f

 previousByte = 0

 # Create an empty byte array.
 buffer2 = bytearray()

 # Loop over all the bytes that were read by retrbinary()
 for b in buffer:

  # Is the byte a LF? Is it NOT preceded by a CR?
  if b == 0x0a and previousByte != 0x0d:

   # Prepend a CR to the LF.
   buffer2.append(0x0d)
  buffer2.append(b)
  previousByte = b

 # Write the modified byte array to the local file.
 f.write(buffer2)

# Retrieve the specified file from the FTP server.
def retrieve(fname):
 global f
 global ftp

 print(fname)

 # Open the local file for writing in binary mode.
 f = open(fname, 'wb')

 # Determine whether the file is binary or text. In this simplified example, a file is
 # binary if and only if the file extension is GIF, JPG or PNG.
 name = fname.lower()
 if name.endswith('.gif') or name.endswith('.jpg') or name.endswith('.png'):

  # Binary file. Don't modify the bytes. Just write them to the local file.
  ftp.retrbinary('RETR ' + fname, f.write)
 else:

  # Text file. Insert CR's as needed before writing to the local file.
  ftp.retrbinary('RETR ' + fname, processBytes)

def main():
 global ftp

 print("STARTING...")

 # Connect to the FTP server. Replace the arguments with your URL, username and password.
 ftp = FTP('ftp.example.com', 'someuser', 'somepassword')

 # List all the files in the current directory, including the file type in the results.
 files = ftp.mlsd('', ['type'])
 for file in files:

  # If it's a file, not a directory, then download it.
  if file[1]['type'] == 'file':
   retrieve(file[0])
 ftp.quit()
 print("DONE!")

if __name__ == '__main__':
 main()

A Python program to back up all repos in a GitHub account

The goal: Automatically find all repos in a specified GitHub account and back them up to folders on the local hard disk.

I've seen instructions for backing up GitHub using Ruby (such as this example) or a shell script (like this one), but I wanted to use Python, which was already installed on my computer.

My environment:

• Windows 7
• Python 3.3.0

Here's the Python code, gitback.py:

import json
import os
import sys
import urllib.request
import zipfile

def main():
 # Change this to your own GitHub username.
 user = 'xxxxx'

 # The repos will be backed up to the directory you
 # specify as a command-line argument.Example: gitback.py c:\gitbkup
 target = sys.argv[1]

 # Delete the target directory if it exists. Then create it.
 if (target != None and target != '' and os.path.exists(target)):
  os.system('rd /S /Q ' + target)
 os.makedirs(target)

 # Get up to 100 repos. If your account has more than 100 repos,
 # increase the per_page value.
 req = urllib.request.Request('https://api.github.com/users/' + user + '/repos?per_page=100')
 resp = urllib.request.urlopen(req)

 # The result is in byte format, so we need to specify
 # the encoding and decode it.
 content = resp.read().decode('utf-8')

 # The result is in JSON format. Parse the JSON.
 data = json.loads(content)

 # Loop over all the repos, cloning each one.
 for repo in data:
  name = repo['name']
  os.system('git clone git@github.com:' + user + '/' + name + ' ' + target + "\\" + name)

if __name__ == '__main__':
 main()

Workaround for problem with fieldset border in IE 10

I discovered a problem with the visual appearance of a page using a fieldset tag in Internet Explorer 10.

Here's a small HTML page that illustrates the problem:

<html>
<body style="background: blue;">
<fieldset style="width: 500px;">
<form>
 <legend>This is my form</legend>
 <p>Enter some text: <input type="text"></p>
 <p><input type="submit" /></p>
</form>
</body>
</html>

Here's how the page renders in IE 10. Note the wide, misaligned right border.

The border looks normal, however, in the other browsers I tested: Chrome 30.0.1599.69 m and Firefox 24.0.

I found a simple workaround, which is to explicitly set the border style for the fieldset tag:

<fieldset style="width: 500px; border: 1px solid white;">

This cleans up the border like so:


And it continues to look fine in Chrome and Firefox.

Of course, you could do the same thing in a CSS file if you prefer:

fieldset {
 border: 1px solid white; 
}

Mod_security causes sporadic "No data received' error on website

I recently received a report from our headquarters office that our corporate website was down. I tried accessing it from my home office and had no problem, but a few minutes later, I too saw the error. In Chrome, it manifested as a "No data received" message. Clicking the "More" button revealed the further description "Unable to load the webpage because the server sent no data" and the error code "ERR_EMPTY_RESPONSE".

I soon discovered the problem was sporadic. Sometimes I was able to view the site's homepage, sometimes not. Sometimes I was able to log in to the site's Wordpress admin page, sometimes not. Sometimes I was able to save changes to a post in Wordpress admin, sometimes not. There didn't seem to be any pattern to which pages produced the error or when.

The site uses WordPress and is hosted on a GoDaddy shared server. Several people have complained about similar problems with WordPress recently (such as here and here), but didn't offer any real solutions.

I contacted GoDaddy support. Their first move was to enable error logging for my hosting account. That led nowhere: although I replicated the problem several times while logging was enabled, and although we waited several minutes for log data to be recorded, nothing showed up in the logs. The support tech concluded that no errors were being logged because the requests were being blocked before reaching the layer where logging would occur.

His next move was to run some tests on GoDaddy's firewall and the servers behind it. These tests showed that everything was operating normally.

After some thought, he concluded that mod_security was to blame. He mentioned that WordPress has recently been the target of a lot of hacking, and that GoDaddy had therefore stepped up security measures, deploying mod_security. Mod_security was mistakenly identifying some, but not all, attempts to access the site, from both headquarters and my home office, as hostile. While we assumed that users in other locations weren't affected, we had no way of knowing.

The tech suggested some workarounds, including clearing cache and cookies, and using a private browsing sessions. Neither of these helped any of the affected users. The one suggestion he offered that did work was to wait.

He indicated that once mod_security identifies a request as hostile, it blocks the sender's IP address for 90 seconds. This explains the sporadic, now-it's-up, now-it's-down nature of the problem. He also indicated that, if several requests from the same IP are flagged as hostile, the duration of the blockage becomes longer. He wasn't able to tell me a formula for how long we needed to wait. I ended up waiting several hours before trying again. At that point, the problem had resolved itself, and it hasn't resurfaced, three days later.

We really have little control over the situation. Because it's a shared hosting account, we don't have access to mod_security settings, or to network and server logs that might help with troubleshooting. The tech indicated that upgrading to a dedicated or virtual dedicated server would give us greater control. So far, we've decided not to upgrade, since it would increase cost, cause some downtime, and take several hours of effort on our part to migrate the site to a new server.

We're keeping our fingers crossed that mod_security decides that we -- and our customers -- aren't evil after all.

Eliminating JavaScript validation errors due to jQuery in an Eclipse dynamic web project

If your Eclipse project includes jQuery files such as jquery-1.8.3.min.js, Eclipse's JavaScript validation may report syntax errors that, although harmless, makes it look as  if your project contains bugs.

Much has been written online about how to resolve this problem, but not all of it is correct. In particular, suggestions involving Properties | Validation | Client-side JavaScript are off the mark, as these settings affect JavaScript within  HTML pages, not standalone .js files.

This comment -- https://bugs.eclipse.org/bugs/show_bug.cgi?id=349020#c15 -- has it right but doesn't give much detail. Here are the exact steps I followed to correct the problem:

1. Right-click the project name in Project Explorer and select Properties.

2. Expand JavaScript and select Include Path.

3. Expand the folder that contains the offending JavaScript files (which will vary depending on how you've set up your project), and click Excluded.

4. Click Edit. Click the Add button next to the Exclusion patterns textarea.

5. Now type in the pattern for files to be excluded, being careful to include the correct path, which will vary depending on how you've set up your project. In my project, the pattern is resources\lib\jquery*.*.

6. Click OK. Click Finish. Click OK. The errors should now disappear.


Behind the scenes, this added the following line to my project's .settings\.jsdtscope file:
<classpathentry excluding="resources/lib/jquery*.*" kind="src" path="src/main/webapp"/>