Robots.txt file is usually placed in the root folder of the website, and for security purposes root folders are more secure than the specific subfolder. Because changes to Robots.txt are managed by SEO personnel, they were often unable to upload the updated Robots file through File manager in Sitefinity, and were dependent on the programmers to make edits to that file.
At Americaneagle.com, we thought SEO personnel should be able to manage robots.txt more easily through the editor in the CMS. Adding a custom configuration was the obvious choice. But the custom configurations that appear under Advanced Settings are limited to simple textboxes. We wanted text area on the custom configuration so it looks more like a text editor and more readable. What we found was the template for advanced setting only supports textbox input and changing that would change it for all the advanced settings.
As an alternative, there is a way to create text area under the Basic setting screen as described here: http://www.sitefinity.com/developer-network/forums/general-discussions-/multisite-custom-field#Zo7eg2ycdkSGElmwBsX5rA
Slightly tweaking this functionality to our needs, we were able create a whole new screen as a Basic setting entry and have a template with the long text field as needed.
With this change we removed the physical robots file, because it is highly possible that if someone is looking at that, they may tend to change that. We used a redirect rule to grab that configuration content and display it whenever the user types in /Robots.txt
Here is the screenshot of how it looks:
/Sitefinity/Administration/Settings/Basic/Robots/
It should be noted that this editor does not validate the contents of the file, it is simply for editing the contents. There are other tools available online for testing Robots file.
Full code is available for download from Gist:
https://gist.github.com/mani-ae/d1fdab9297264d0522063e131487a52c
Here are code snippets:
Americaneagle.com can help you in implementing this Robots file editor in Sitefinity, which will help SEO personnel to manage using the visual interface. Contact us for more details.
Contact Us Today!