So my carefully planned afternoon of work on client projects took an unexpected turn today when I opened up an email from somebody I had met at my Friday BNI meeting. He had googled my business’ name (Jenn’s Web) and come up with a troubling result.
ARRGH! How did that happen?? Suddenly I realized that the past few months of constant client work had taken a toll on my own site (NOT that I’m complaining!).
At some point during the summer, I had re-pointed my new domain name, Jennsweb.net to my hosting account and when I entered http://jennsweb.net in my browser’s address bar, my site came up (albeit with its original hosting server name of jennmearswebde.wpengine.com), but I just hadn’t had any time to really deal with the ins and outs of properly setting up my domain and now the chickens had come home to roost.
Even though my clients often say things like “Wow, how did you learn all this technical stuff?” and sometimes I have managed to surprise myself with what I can figure out, terms like “robots.txt” and “.htaccess” still make me want to huddle up in a pillow fort for the rest of the day. Mainly this is because 1: I usually don’t have to deal with things like that very often unless something is very wrong on a site, and 2: It involves mucking around with things like “root folder permissions” and “shell commands”.
Taking a deep breath, I put aside my “to do” list for the day and dove in. First, I clicked on the words “Learn More” in the search result. That brought me to this page:
How I Fixed This
As much as we all depend on Google to guide us to what is safe and informative on the web, sometimes I get a little leery of falling into a “Google Hole”. This is when you start with one page in Google’s many, many subdomains and have to click on another page, and then another and before you know it, you are in a maze of browser tabs and links so deep you forgot what you even clicked on to begin your journey.
My first instinct was to check in with my hosting company to see if there was some (hopefully) obvious reason my site was missing its robots.txt file. “But wait,” I thought, “What if this is somehow tied to the domain name re-pointing? “If I switch my domain name in my site’s General Settings to jennsweb.net won’t that affect my previous links? And hey, what about adding an SSL certificate to my site while I’m at it?”
I wrote down everything I wanted to try and figure out and opened up a chat with WPEngine. These guys are amazing. I immediately got in touch with a very nice young man named Clay and I poured my heart out. “What should I do first?? How do I upload a robots.txt file?? How the heck do I even create a robots.txt file???”
Clay sent me a couple of links and then waited patiently as I skimmed through them to figure out what I needed to do. I listed out the tasks I wanted to do, from fixing the robots.txt issue to installing an SSL certificate. Turns out, I actually had the order of things more or less correct.
Fixing your domain url after re-pointing a new domain name
1: Get an SSL certificate installed on your site.
Some hosting companies charge an extra fee for this. Luckily, my hosting company will provide one as part of the hosting package. It’s worth doing. Here’s why. I wanted to get the proper syntax on the domain name before I made it official.
It’s a good idea to wait until your SSL certificate has been completely installed by your hosting company before going to Step 2.
2: Change your WordPress Address and your Site Address in your dashboard’s General Settings.
This is where your WordPress site decides what to call itself. I needed to change it in this case from “http://jennmearswebde.wpengine.com”, the default url issued by my hosting provider, to “https://jennsweb.net”. What’s the difference between your WordPress Address and your Site Address you ask? Basically, the WordPress address is for the dashboard of your site and the Site Address setting is for the “front-end” or, what your site’s visitors see.
Here is the updated setting for the site’s WordPress and Site addresses
TIP: Make sure you have your WordPress admin username and password stored somewhere handy. Once you change this setting, you will need to log back into your site due to the name being changed.
3: NOW, you can upload your new robots.txt file.
The reason I embarked on this journey in the first place! One thing I have learned in dealing with Google, it really helps to have all your ducks in a row BEFORE you submit anything to its Search Console tool. Otherwise, you try, it fails, and the great and powerful Google steamrolls on, leaving your site in the dust until it rolls around again.
How do you create a robot.txt file? What the heck IS a robots.txt file??
Googlebots and other types of webcrawlers, otherwise known as search engine spiders, are much like actual living spiders. They don’t really think too much. They simply sense movement nearby, go towards it, and try to sense if it’s prey or not. If it is, they pounce and feed. If not, they simply ignore it. Your robots.txt file is supposed to act as an instant indicator if your website is “prey” i.e. available to be indexed, (or not). So, you need to create a text file that contains some code that will let the webcrawler know which parts of your site you would like to show up on a Google search.
Before you create the file, make sure you can upload it to the root folder of your site. Your root folder is the topmost folder available to a browser. A lot of web hosting companies designate all your files to a root folder labeled “public_html”. If you aren’t sure which is your root folder, you can ask your hosting company.
WPBeginner has a very helpful article on what you need to include in your robots.txt file. You definitely want your site’s pages and posts and images to be searchable, but probably don’t want your plugin and theme files to show up on a SERP, so this is where you want to tell search engines exactly what is up for grabs.
For example, my robots.txt file ended up looking like this
4: Test and Submit your robots.txt file in Google.
To make sure that your new robots.txt file is functioning properly, Google offers this tool to test out your file. When you click on “Open Robots.txt tester” it will open a new tab that looks like this.
Confession time: my first attempt at creating a text file contained some hidden formatting code and my initial submission resulted in a list of errors. Google’s tester tool is pretty helpful in that it allows you to make the edits directly on the results page and then download the corrected file to then re-upload and test again. Here’s some guidance on creating a truly plain text file.
If you have more questions, check Google’s robots.txt faqs here.
So, what I thought was going to be a quick little file upload turned into a lot more. Sometimes a trip down a “Google Hole” is well worth the extra time.
The post Fun With Robots appeared first on Jenn's Web.