Matigo Goes Ubuntu

Back in the day when I used Windows on a desktop, one common problem that I struggled with was having an abundance of storage devices connected to the machine. There were typically four or more drives inside the case, plus two or three USB devices, then network storage on top of that. At the day job in 2010 this sort of problem was seen again, and each drive would have its own dedicated purpose as a result. C: was for the system and applications. D: was for code. E: was an SSD, so dedicated to local databases. F: was a dumping ground for everything, including a copy of my iTunes library. Network drives started at Z: and went backwards, except for the Windows Home Server backup machine, that was A: …

What a pain in the bum that was, especially when one drive started to run out of space. Then you were required to make a decision:

1. move a bunch of data around
2. accept that some of the data was in one place, and the rest in another
3. drop a lot of money to get a nice RAID card and spend a weekend watching Windows copy data from one place to another

We've probably all done the first two items, and I've done the third on a number of occasions as well. Luckily, with Ubuntu (and other modern OSes with Logical Volume Managers), this is much, much easier to deal with. We can connect a lot of storage devices to our machines and have them appear as a single logical unit, taking the pain out of trying to organize where files will go on our computers. I did this with the two 512GB SSDs on the work Lenovo, and now I'm going to show you how I did it.

Note: In order to add drives to an existing LVM, they must be completely empty and have no partition information on them. If you want to add a drive that already has a bunch of information to the storage pool without losing any information, this is not the process you're looking for.

Note 2: Using an LVM or RAID is no substitute for a proper backup mechanism. Please make proper backups of your data in the event something really, really bad happens to your computers.

For the sake of this walkthrough, I'm going to use a virtual machine with virtual drives. For this example, we have a computer with a 128GB primary hard drive and a 2.0TB device is being added to the existing storage pool.

First things first, grab your Ubuntu MATE 16.04 Live DVD, because you're going to be doing everything from within a live session. This is mainly because OSes generally do not like big changes happening to the boot partition during operations. Some people will say that they've resized their partitions while live, but this doesn't mean it's the right way to do a thing.

Once the Live MATE finishes booting, we need to make a physical drive. To do this, we need to know the device name of the new disk. Open up the Disk Management tool and find the new device. The partition will likely be unallocated or, if it came pre-formatted from the factory, NTFS. Make note of the device name, in this case /dev/sdb and open a terminal session.

In the terminal, type:

1. pvcreate /dev/sdb

Be sure to replace sdb with your proper device name.

A moment later you will notice that the hard disk now shows LVM2 PV. From here, let's grab a handy little tool called system-config-lvm. Launch a terminal and type:

1. sudo apt-get install system-config-lvm

If nothing is found in the repository under that name, then you'll need to add a repository that does have the tool and update, like so:

1. sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe"
2. sudo apt-get update
3. sudo apt-get install system-config-lvm

After installation you can find "Logical Volume Management" in Applications ⇢ System Tools. Launch the application and you'll see something like this:

Click the unallocated volumes and then choose your new device. You'll see a red bar denoting your drive, and three buttons underneath. Since we're adding this drive to an existing storage pool, click "Add to Existing Volume Group". You'll then see a window pop up asking which volume group to add it to. As this example only has one, we'll add it to that.

Which gives us:

And now, at last, we can expand the existing volume from about 123GB to almost 2.2TB. Expand the "Logical View" and choose the volume you're expanding. In this case, it's "root". You'll see a single blue cylinder representing the drive. Click the "Edit Properties" button to bring up the volume attributes. Near the centre you'll see a handy little button that says "Use Remaining", which will make the volume as large as it can be with the devices. Once done, click OK and the software will work its magic. Reboot back into your proper OS and confirm you now have what appears to be a single drive that is much bigger than before.

The tools to do this have come a long way in the last decade. I remember doing this with other Linux flavours in the early 2000s and it was always a nerve-wracking experience.

Yesterday I talked about breaking all sorts of rules and replacing both the hardware and the software on a company-owned Lenovo W541. All in all, I'm happy to have turned the notebook into a useful piece of kit, but I still need to use the Windows installation my employer provided in order to access certain network resources. Dual-booting is not an option, so this means using a virtual machine. The question now is how we can take a company-supplied Windows installation and turn it into something that VirtualBox can use.

Oddly enough, the process is not as difficult as you might expect.

Disclaimer: Just because we can do a thing does not mean we should do a thing. I'm in a rather unique position with my employer which grants a certain amount of freedom and flexibility with regards to computing. Not everybody is this lucky. Be sure to consider any possible consequences of your actions if you plan on doing things to hardware you don't own. That said, if you're doing this to machines you have complete control over, I hope this document provides some value.

Mark Russinovich over at Microsoft released a very, very useful SysInternals tool called Disk2VHD and you can get it for free from the Microsoft site. Do yourself a favour and only get it direct from Microsoft. It's pretty small, coming in at under a megabyte and it has everything you need to make your current Windows installation into a virtual machine image that is readable by VirutalBox (and other virtual machine engines). You will need to run this from within Windows, and It needs both administrator access and a bunch of free disk space. When you make an image of your current hard drive, you will need at least that much free space for the destination file.

When you go to make an image, be sure you also include the boot volume and uncheck the "Use Vhdx" checkbox. I didn't do this the first time, and it meant having an non-bootable .vhd file. I then had to boot into Windows and start all over again. I suggest playing it save and copying all of the strangely-named volumes, plus the Windows partition. You probably won't see the ¥ symbols in your volume names if you are in an English speaking country. Windows in Japan is really, really hard to look at.

Once you're happy with all of the settings hit the "Create" button and let the computer do its thing. Depending on the size of the image and the speed of your drives, the process can take anywhere between five minutes and many, many hours. Doing this before going to lunch is probably a good idea.

Once your file is made it's time to bring it over to Ubuntu. Unfortunately, it's not a simple process to get it working.

Converting the .vhd to a .vdi

VirutalBox can use .vhd files, but I opted to convert the image to a .vdi so that I could manage the file with VBoxManage. It's a proprietary format, but the advantages of using VirtualBox's built-in tools to resize or compress the images outweighs data portability for me. Also, I don't use other VM software, so cross-compatibility is a non-issue at this time. If you're not interested in converting the file to a .vdi, jump down to "Using the .vdi" and mentally replace the file extension with .vhd in your head.

Now, unfortunately, I had a bit of trouble converting the image with VirtualBox 5.0.16, and was forced to temporarily install 4.3. If, like me, you have VirtualBox 5.0 installed, you can uninstall it with the following commands:

1. sudo apt-get remove virtualbox-\*
2. sudo apt-get purge virtualbox-\*

From here you can install 4.3.x. Now you'll also need to make sure that your machine has the current kernel headers installed, otherwise VirutalBox will complain when you try to convert the file. You can do this from the terminal with the following command:

1. sudo apt-get install linux-headers-$(uname -r)

Now, at last, you can convert the file. From the terminal, type:

1. VBoxManage clonehd /path/to/VHD /path/to/new/VDI --format VDI

This process might take a while, so give the computer some time. This would be a good opportunity to get up and walk the dog or use the washroom.

Using the .vdi

To use the .vdi file we need to first create a new virtual machine image. For this I decided to create the image in VirtualBox 4.3 in the event something went wrong and I'd have to try again. Luckily, that didn't happen. Either way, here's the process. First, create a new Virtual machine and give it a good, descriptive name.

From there, click "Next" and set the amount of memory to dedicate then "Next" again to show the storage options. Here you'll want to choose "Use an existing virtual hard disk file" and select the one you just created.

From here you can hit "Create" and voila! You are now the proud virtual operator of an exact image of your Windows machine! Be sure to configure the hardware to include just the things you need, set up some shared folders so that you're not running Dropbox in the VM and on the host, and go to town.

Wait … Why?

One question that some people have asked me is why I would go through all this trouble. I spend a great deal of my day at the office using Windows, so why not just leave the computer as it was and run everything in Windows?

A valid question. My reasoning is simple. When bad things happen, Windows becomes a liability. Did IT push out an update that messed up your computer? No problem. Grab a back up of your VM (you do keep backups … right?) from the night or week before. So long as all of your important data is not kept on the Windows VM, it's all good. You can be up and running again in less than five minutes. Did you somehow contract a virus? Same thing. Kill the VM, copy a backup. Did a recent Windows Update turn your once-speedy computer into a sluggish pile of plastic and silicon? Once again, kill the VM and copy a backup.

I really don't like downtime, and using VMs allows me to very quickly recover from loss or throw away an OS update if something goes awry. Even if the Ubuntu host were to suffer from a serious failure, I'd be able to bring the VMs from the backup over to another computer running VirtualBox and (mostly) pick up where I left off within minutes. You can't tell me that any corporate IT department could get you a replacement computer or roll back a bad update in less time than it would take to copy a .vdi from an external or networked hard drive.

So there we go! The work Lenovo is Ubuntu under the covers, and has the "approved" Windows image available any time its needed with far more versatility than most IT departments can offer. I'd call this a win-win.

Earlier this month I was given a new notebook at the day job to develop our next-generation education-supporting software with. All in all, it's a potent machine with a quad-core Intel i7 processor and a beefy nVidia K2100M video card. There were just a few problems with it. They included:

• a 5400rpm hard drive
• just 8 GB of RAM
• a severely locked down version of Windows 7
• errors galore when launching any application
• sluggish, sluggish performance

This notebook, a Lenovo W541, is supposed to be either the second-most — if not the most — powerful computer I've had the luxury to use. When the IT manager asked if the machine would be "acceptable to work on", I nearly leapt from my seat because it's more than acceptable. Heck, it's downright generous considering how I'm writing software in C#. I don't really need the amount of power they've provided. That said, for the configuration I was given, it's like buying someone a Ferrari with a speed governor installed and set to 45kph. Yeah, it's nice … but it's painful, too.

So, not wanting to feel frustrated every moment at the office, I decided to break some rules, spend some cash I earned with some freelancing on the side, and upgrade some internals. Here's what I installed:

• 32GB DDR3L 1600 MHz RAM (Transcend TS1600KWSH-16GK x2)
• 512GB SATA III 6Gb/s M.2 SSD (Transcend MTS400)
• 512GB SATA III 6Gb/s 2.5" SSD (Transcend SSD370S)

It's a wonderful bit of kit, and I was able to get it all from Amazon for a decent price. The nice thing about this particular Lenovo is that is has a pair of M.2 slots, one of which is used by the WiFi card, and four SODIMM slots. Now, a word of warning to anyone with a W540 or W541 who wants to fill all four slots: only the quad-core processors (the i7 4xxx-QM series) can support four SODIMMs. Dual-core processors will only support slots 0 and 1 despite having four slots next to the 2.5" drive bay. That said, some examination of the CPU socket reveals that it may be possible to replace the processor yourself. I haven't physically verified this by taking the entire bottom of the notebook off but, from what I could see, it certainly appeared to be possible. That would make this notebook the first I've ever had where it was possible.

Back to the task at hand, though …

Installation of the hardware was a snap. The bottom panels, with some sliding around and nervous prying, open right up to reveal the necessary slots. Everything clipped in within seconds, and I was booting the Ubuntu 16.04 MATE Beta 2 Live DVD in no time flat.

The reason I opted to go with 16.04 instead of 15.10 had a lot to do with driver support. Having read through a lot of posts all over the Internet where people explained some of the difficulties they had installing Ubuntu on the W54x model Lenovos, I wanted as little pain as possible. People were saying good things about 16.04, so I threw caution to the wind. If it wasn't going to work, then 15.10 would also be possible, but it needs a little bit of care so that you don't brick your motherboard. 12.04 LTS is technically supported and certified, but there's a lot you lose by going with that release. So, 16.04 seemed to be the most logical version to go with.

The 16.04 MATE installation process was incredibly painless. I told the software where to put the OS, configured some partitions, and let the computer to all the work. Updates were pulled down during installation, and all the important bits "just worked"™ like you would expect it to in 2016. After installation there were a few extra things that needed some software, the fingerprint reader, video card, and power management systems most notably, and this was easy enough from the terminal.

To install the fingerprint reader drivers and software:

1. sudo add-apt-repository ppa:fingerprint/fingerprint-gui
2. sudo apt-get update
3. sudo apt-get install libbsapi policykit-1-fingerprint-gui fingerprint-gui

To install the power management drivers:

1. sudo add-apt-repository ppa:linrunner/tlp
2. sudo apt-get update
3. sudo apt-get install tlp tlp-rdw
4. sudo apt-get install tp-smapi-dkms acpi-call-dkms

Then the nVidia drivers. Before installing these, it's a good idea to eliminate any existing ones that might have been installed beforehand.

1. sudo apt-get remove --purge 'nvidia*'
2. sudo apt-get remove --purge xserver-xorg-video-nouveau

Then:

1. sudo add-apt-repository ppa:graphics-drivers/ppa -y
2. sudo apt-get update && sudo apt-get upgrade -y
3. sudo apt-get install linux-source && sudo apt-get install linux-headers-$(uname -r)
4. sudo apt-get install nvidia-362 nvidia-settings

Once all this is done, restart the machine, and you'll be good to go. Under Windows 7 I was able to squeeze just under 4 hours out of the battery while developing in Visual Studio. Under 16.04 MATE I'm able to get about 5 hours on a single charge. Still not as great as a MacBook Pro, which can go for close to 12 hours without touching an AC adapter, but not bad considering it's a big, beefy Lenovo workstation.

Over the last few months I've been making the transition from OS X back to Linux. I go over a number of reasons for this change on the Changing Platforms podcast with my friend Keita, so won't really get into them at this point in time. Instead, I'm happy to report that, as of March 31st 2016, I am now using Ubuntu exclusively for about 60% of my computing time. More than this, though, I've broken all sorts of rules and replaced Windows 7 on my work notebook with Ubuntu Mate 16.04 (Beta 2) and couldn't be happier. It's a very solid operating system, and it brings the hardware to life in ways that Windows never could.

As this is the first post on my new blog dedicated to my time with Ubuntu, I thought I'd go over some of my experience with the software as well as my recent impressions of the human-friendly operating system.

The first time I used Ubuntu was shortly after the release of 6.06, also known as Dapper Drake. It's hard to believe that a decade has passed. At this time I was working as a developer in Vancouver and was looking for an alternative to Windows XP, which seemed to struggle on workstation-grade hardware every few months. I wanted something different. Something that I could rely on. Something that I could reach right into and change if needs be. I had experience with RedHat, SuSE, and Slackware, but was intrigued by this new flavour with the African name. I ordered an installation disc and, soon after installing it on a simple desktop I had sitting idle, I was impressed.

Interfaces on Linux have always had reminded me a great deal of dosshell.exe or Windows 3.1. Ubuntu was different, though. It looked like a cross between something discussed in Cupertino, designed in Redmond, and coded at MIT. It wasn't beautiful, but it was rock solid and got the job done. I could (and did) write software for the platform without swearing as often as I did at the day job, and never once did I need to reinstall the operating system because an update killed something important.

After moving from Canada to Japan, Ubuntu played an important role at my house. I needed a network server that would happily store lots of data, grow with ease, support UTF-8 filenames, transfer files throughout the day, and run on an "ancient" Pentium III 500MHz. Windows XP could certainly do this for a while, but the machine would get awfully hot in the summertime and freeze. I needed something that wouldn't tax the system so much while also providing all of the functionality I was looking for. Ubuntu Server seemed like a perfect solution, and it was. I installed 8.04 LTS, put the simple server in the back of a closet, and walked away for nearly two years. The only time I even looked at the machine was when another hard drive needed to be added to the storage pool.

Every NAS since has run Ubuntu Server, and the machine that currently serves files and manages the printer at home runs 12.04 LTS with just over 42TB in its storage pool. It's rock solid … albeit a little slow given the age of the hardware.

On the desktop, I used every version between 8.04 LTS and 10.04 LTS and enjoyed many of the refinements that had come with each release. My primary use case was rather simple, though. Blogging, basic image editing, torrents, listening to music, and watching documentaries. Very light stuff, indeed. But in the summer of 2010 my wife and I moved from rural Japan to Tokyo where I had accepted a job at a tech startup. Windows was the platform of choice there, and I became invested in that platform once again … until 2012 when I ditched Windows for its insufferable incompetence when dealing with multiple languages. It was then that I moved to OS X. I learned quite a bit about design from using software built for Apple's ecosystem, but it's just not for me. Not anymore, anyway. I need something that doesn't try to do so much in the background. I need something that doesn't try to send so much information to some server somewhere "in the cloud". I need something that doesn't try to sell me upgrades at every turn.

I need to return to something that, aside from Unity, rarely frustrated me. So earlier this year I made the decision to move back to Linux and, with some quality second-hand hardware, I've decided that Ubuntu 16.04 MATE is the way to go. Why MATE? Well … I just think it looks better.

The OS is a joy to work with on a MacBook Pro, and I've even put it on my work notebook, a Lenovo W541 — a machine that I also took the liberty of pimping out a bit to get the most from the incredible hardware. That said, while I've been using Ubuntu Server for almost a decade, I've been out of the loop with the desktop version of the platform. This blog will act as a place where I write my How To's, reviews, goals, and objectives with Ubuntu in the coming months and — hopefully — years. My software needs have changed quite a bit since 2011, and I'll need to find or create tools that will enable me to be just as productive on this platform.

As I said earlier in the post, I'm spending 60% of my time in Ubuntu when working with computers. It's the last 40% that will be a little tricky and require some serious effort. But where there's a will, there's a way. Hopefully some of what I do can be shared with the community to help others who might seek to accomplish similar goals.