Programmer's Guide to the Webif
From X-Wrt
The OpenWrt web interface is based on a set of shell and awk scripts and the form processing is done with haserl. It uses the Busy``Box HTTPD server. And is commonly called Webif. Webif^2 is based on Openwrt's web administration tool, Webif.
The Programmers Guide to Webif2 is the only documentation available. It is very much an early work in progress, edited by numerous people, mostly oxo-owen.
Contents |
What is a webif page?
A webif page is essentially an HTML page with embedded shell script. Core functions, like the page header/footer and settings forms are implemented by an AWK back-end. For example, see /usr/lib/webif/form.awk, which implements 'display_form' calls in the webif pages.
Example: info.sh
#!/usr/bin/webif-page <? . /usr/lib/webif/webif.sh header "Info" "System Information" "@TR<<System Information>>" '' '' # A lot of shell code .... footer ?> <!-- ##WEBIF:name:Info:1:System Information -->
- The first line tells us that the program that is called a binary program /usr/bin/webif-page - webif-page is a suprisingly small in c : see latter in conection with translation
- The second line <? is a bit of magic so we can combine html and shell scripts - Its sister, ?> at the end finishes that magic show.
- The third line means we have some nice library functions that can be drawn on in /usr/lib/webif - there are more: have a look.
- The fourth line gives a title - @TR: see latter in connection with localisation (TRanslation)
- Then a lot of nice shell scripting - header and footer are NOT football terms but examples of the nice functions we can re-use
- The file closes with a cryptic ##WEBIF: which is used as housekeeping for the menu structure of Webif. /www/cgi-bin/.categories contains the order of categories.
##WEBIF:category:Info ##WEBIF:category:Graphs ##WEBIF:category:Status ##WEBIF:category:Log ##WEBIF:category:- ##WEBIF:category:System ##WEBIF:category:Network ##WEBIF:category:VPN ##WEBIF:category:HotSpot ##WEBIF:category:- ##WEBIF:category:Logout
Example: Hello world page
The classic example - or do nothing with style
- add to .categories
##WEBIF:category:HelloWorld
- cp info.sh to helloworld.sh in cgi-bin/webif
- alter the corresponding lines
header "Info" "System Information" "@TR<<System Information>>" '' '' ##WEBIF:name:Info:1:System Information
header "HelloWorld" "Hello World" "@TR<<Hello World>>" '' '' ##WEBIF:name:HelloWorld:1:Hello World
Please remember that header text has to match the ##WEBIF line.
Congratulations, you just made your first do nothing Webif module. Point your browser at your box (maybe reload with no cache) and see your own greeting. You can now basically make any status page you want.
If you want, instead of editing .categories, the category can be defined by your script. It will be ordered last. For example:
<!-- ##WEBIF:category:HelloWorld ##WEBIF:name:HelloWorld:1:Hello World -->
Basic page limits
The contents of the page can be any combination of the shell script combined with a HTML text (see <? ... ?>). The longest line that Webif2 can process is 2048 bytes long.
Save Apply Clear Review Changes
We'd better look at some of those pre-defined buttons:
Save Changes
- returns a FORM_submit to your script so you can: if !empty $FORM_submit;then "some code" fi
Apply Clear Review Changes
- come from footer in webif.sh
- Apply Changes calls /www/cgi-bin/webif/config.sh with FORM_mode=save, config.sh will then execute /usr/lib/webif/apply.sh
- Clear Changes calls /www/cgi-bin/webif/config.sh with FORM_mode=clear, config.sh will delete all temperary files in /tmp/.webif/ and /tmp/.uci/
- Review Changes calls /www/cgi-bin/webif/config.sh with FORM_mode=review, config.sh looks for and displays /tmp/.webif/config-*, /tmp/.uci/*, and will display the file name of any edited files /tmp/.webif/file-* and /tmp/.webif/edited-files/*
apply.sh
/usr/lib/webif/apply.sh is where all the temporary files are parsed and the settings from them get saved, also this script handles any necessary restarting of programs. apply.sh also looks for /usr/lib/webif/apply-<your name>.sh which means that you can add your own ad-hoc pages.
The processing sequence of a page
You will learn the chain of applications and events that at the end form up the Webif2 page.
- httpd service receives the request from a browser.
- If the access to the page must be authenticated, it requests login credentials.
- It forks another process for creating the response. When a url contains "cgi-bin" it is assumed to be a cgi script. The server changes directory to the location of the script and executes it after setting QUERY_STRING and other (CGI) environment variables.
- The CGI application is invoked with pipes for stdin/stdout. httpd forwards both QUERY_STRING variable and POST data to the CGI application. In our case, it executes #!/usr/bin/webif-page.
- webif-page is a small and fast programm that is responsible for loading translation files, forwarding the script to haserl and translating the haserl's output before sending it to a client.
- haserl is a very small CGI wrapper that enables a shell script to be embedded into html documents.
- It transparently parses POST and GET requests and places form elements as FORM_variable=value pairs into the environment for subsequent use by the script. It scans the environment for HTTP_COOKIE, parses the contents and places it in the local environment. It reads and parses standard input depending on the REQUEST_METHOD variable and places its contents in the local environment. It prints the contents of the script on standard output (stdout), and conditionally interprets a text within <? ... ?> as a shell script.
- It then parses the code blocks from raw text. The raw text is sent to stdout (webif-page). The code block is sent as a script to the subshell with the appended command echo $? >&5. Again any data on the standard output is sent to the standard output.
- After the last text has been processed, the subshell is closed and the haserl interpreter terminates and returns the control to the calling application (webif-page).
- Note: The file descriptor 5 is reserved for a special communication channel with the parent process.
- The webif-page application terminates and returns the control to the calling application (httpd process).
- Note: Please notice that the TRanslation is performed after all other chained applications has received the script and has substituted variables.
- The httpd's process sends all data remaining in the output buffer (stdout) to the client and terminates.
Localization
Thanks to the originators of the webif-page, CGI wrapper the page contents can be localized. Oh, do not expect the gettext's comfort. It is a simple one-to-one text replacement.
How it works
Localization is accomplished by a post-processor built into webif-page which replaces all '@TR<<symbolname>>' variables with the corresponding symbol value in the currently active language symbol file. If no symbol is found, the symbol name itself is used for the text. Therefore, simply using many @TR<<text>> macros for strings is all that initially needs to be done to make a webif page ready for localization.
Translators can later add the symbols to the localized symbol file.The localized symbol files are, as of White Russian RC6 and newer, stored in separate packages instead of all being included in the base webif set.
Webif2 Language Setting
It uses the new UCI configuration file /etc/config/webif, section "general" and a value of the parameter "lang" (from r2736, older releases queried the nvram value of the field "language").
The value of the "lang" parameter should be the language code according to the ISO-631 or more precisely ISO-639-2, e.g.: de for German, cs for Czech and so on.
Where are the language files stored?
The language directory is /usr/lib/webif/lang/<language code>/. (See previous section).
The basic webif2 language file is common.txt. Webif-page accepts also <current-page-name-without-extension>.txt in the language directory. Which is a big help. So understand "common.txt" as it is and try and reuse text. Page dependent language files can be added without changing common.txt
Format of language files
The format is simple:
original term => translated term
where 'original term' is the text from the page source in US English, ' => ' is delimiter (notice the spaces) and 'translated term' is the text translated into the particular language.
The '#' mark in the first column means a comment, this line will be skipped over by webif-page, just as any blank line and lines without the ' => ' delimiter.
The language file format looks similarly to this example:
# this is the example of translation from US English to US English Status => Status DHCP Clients => DHCP Clients status_leases_dhcp_leases => DHCP Leases status_leases_MAC => MAC Address status_leases_IP => IP Address status_leases_Name => Name status_leases_Expires => Expires in
The old language files comming from OpenWrt were encoded in ISO-8859-1 using HTML escapes. This way the page encoding could be ISO-8859-1 and it was possible to see the full Unicode repertoire of characters.
The language files encoding was changed to UTF-8 in the r3165-r3170. See: UTF-8.
Format of the @TR<<>> macro
As you have seen in previous sections, the simplest form is the English word surrounded by the macro delimiters: @TR<<Apply>>.
Because there is the need to translate the particular word or expression into different forms, you can use the translation symbol: @TR<<this particular symbol#Apply>>.
There is another delimiter that can be used in case where '#' character is not available: @TR<<this particular symbol|Apply>>. Please notice that the '|' character is used in form functions as the field delimiter.
Seemingly hidden translations
The @TR<<>> macro is used in many levels to ease the page authoring. You are not required to use it in several form elements, some webif.sh functions provided by the browser.awk, categories.awk, common.awk, editor.awk, form.awk, functions.sh, subcategories.awk, validate.awk files in /usr/lib/webif/.
Look at the following automatically translated elements enumeration:
- categories and subcategories; they appear in the first two parameters of the header() function and in the ##WEBIF:name... line.
- standalone button(name, caption) function
- helpitem and helptext form elements
- un/selected option form elements
- listedit form elements
@TR<<>> macro's quirks
The @TR<<>> macro behavior is predictable but you can experience seemingly wrong output in some cases. Try to avoid them as much as possible.
- The translation symbol cannot contain shell variables.
- The reason is simple, the webif-page CGI wrapper processes the translation after the variables substitution. Simply the translation symbol containing a shell variable will never match the text in the translation file. There is no cure, this is the way how it works.
- The text for translation should not end with the [X]HTML tag.
- The webif-page function searches for the end tag '>>' so that it will match wrong pair of the closing brackets. The fault is emphasized in the example: "@TR<<...see page</a>> >". The respective translation would look like "...see page</a" instead of the "...see page</a>" as you would think.
- You can simply avoid it by rewording the text or placing the [X]HTML escape at the end. You cannot use the simple space or tabulator, they would be stripped, see the next paragraph.
- The translation symbol cannot be surrounded by spaces.
- The webif-page translation functions strip all whitespaces at both ends of the translation symbol (do not ask me why). The most frequent place for this trouble is the form button's text (value=" Push me ").
- You can avoid it by using the @TR macro only on the text without spaces or if it is unavoidable, you can again use [X]HTML escapes ( ), e.g.: value=" @TR<<Push me>> " or value="@TR<< Push me >>".
- The translation symbol cannot contain the characters '#' and '|'.
- These characters are used as delimiters for splitting the translation symbol and the translated text and as a delimiter between form fields. The only exception is that you use the translation symbol, example @TR<<symbol_of_whatever#Do whatever # you wish>> instead of @TR<<Do whatever # you wish>>.
- The escaped sequences in the translation macro must be unescaped in the translation.
- The webif-page is postprocessor and no shell evaluation occurs after the translation. You can see these escapes mainly in the href tag. Therefore the translation symbol containing for example <a href=\"http://.....\"> should be translated like <a href="http://.....">.
- The translation macro must occur on one line.
- The @TR<<>> macro must start and end on the same source line. It is the webif-page's limit.
- The longest @TR<<>> macro including the rest of the line must be shorter than 2048 bytes.
- The webif-page has the fixed limit for reading the haserl's output. It is the webif-page's limit.
- The longest translation, the full "symbol => translation", must be shorter than 2048 bytes.
- The webif-page has the fixed limit for reading language files. It is the webif-page's limit.
- You cannot use the '?' character before the '>>' closing tag.
- haserl CGI Error, Error: Found ?> tag before <? tag near line xx
Differences between languages
- The main problem is that the order of words in other languages is almost always different to Anglo-Saxon ones. It is almost impossible to translate hacked sentences without the translation being choppy.
- The typical example is the nesting of texts. e.g.: "Do you want to" action "\"" pkg "\" package". Such sentences become a nightmare for translators.
- The second serious problem is that one word can be translated to several forms regarding the gender and inflect.
- It is not possible to use the simple form of @TR<<Enable>> because the same word can appear in another page in different context where the other form would be necessary.
Guidelines for page authors (please)
- Try to provide as much translatable texts as you can.
- Supply translation symbols to all translatable fields.
- They enable to translate the same word into different forms when unique.
- For example: @TR<<sp_runproc#Running processes>> instead of @TR<<Running processes>>
- Try to avoid all known issues (see #@TR<<>> macro's quirks)
- Try to please the users in other languages #Differences between languages)
Programmer environment
httpd and cgi
The OpenWrt web interface is based on a set of shell and awk scripts and the form processing is done with haserl. It uses the BusyBox HTTPD server.
FootNote: From OpenWrt's Faq: still investigating
root@oxo-t:/www/cgi-bin/webif# webif-page This is haserl version 0.8.0 This program runs as a cgi interpeter, not interactively. Bug reports to: Nathan Angelacos <nangel@users.sourceforge.net> root@oxo-t:/www/cgi-bin/webif# haserl This is haserl version 0.8.0 This program runs as a cgi interpeter, not interactively. Bug reports to: Nathan Angelacos <nangel@users.sourceforge.net>
CGI Environment Variables
Older versions of httpd returned:
AUTH_TYPE='Basic' COOKIE_style='null' GATEWAY_INTERFACE='CGI/1.1' HASERLVER='0.8.0' HTTP_COOKIE='style=null' HTTP_REFERER='http://192.168.1.1/cgi-bin/webif/info.sh' HTTP_USER_AGENT='http://192.168.1.1/cgi-bin/webif/info.sh' PATH='/usr/bin:/bin:/usr/sbin:/sbin' PATH_INFO='' PWD='/www/cgi-bin/webif' QUERY_STRING='' REMOTE_ADDR='192.168.1.2' REMOTE_PORT='45230' REMOTE_USER='admin' REQUEST_METHOD='GET' REQUEST_URI='/cgi-bin/webif/info.sh' SCRIPT_NAME='/cgi-bin/webif/info.sh' SERVER_PORT='80' SERVER_PROTOCOL='HTTP/1.0' SERVER_SOFTWARE='busybox httpd/1.35 6-Oct-2004' SESSIONID='63de45d73ac1'
Newer (around 1.4.0?) versions of httpd return:
AUTH_TYPE='Basic' COOKIE_style='null' GATEWAY_INTERFACE='CGI/1.1' HASERLVER='0.8.0' HTTP_COOKIE='style=null' HTTP_REFERER='http://192.168.1.1/cgi-bin/webif/info.sh' HTTP_USER_AGENT='Mozilla/5.0 (X11; U; Linux i686; cs-CZ; rv:1.8.0.6) Gecko/20060822 Fedora/1.5.0.6-2 Firefox/1.5.0.6' PATH='/usr/bin:/bin:/usr/sbin:/sbin' PATH_INFO='' PWD='/www/cgi-bin/webif' QUERY_STRING='' REMOTE_ADDR='192.168.1.2' REMOTE_PORT='36682' REMOTE_USER='admin' REQUEST_METHOD='GET' REQUEST_URI='/cgi-bin/webif/info.sh' SCRIPT_FILENAME='/www/cgi-bin/webif/info.sh' SCRIPT_NAME='/cgi-bin/webif/info.sh' SERVER_PORT='80' SERVER_PROTOCOL='HTTP/1.0' SERVER_SOFTWARE='busybox httpd/1.35 6-Oct-2004' SESSIONID='8a246114082'
ash - the shell
$(<file) doesn't work $(cat file) does - apart from that very like bash but there are probably more gotcha's
Developing and Testing Pages
Consider the 'nano' text editor, which is available as a package for OpenWrt (see the backports repository for White Russian).
Better, mount a network share on your router, then symlink /www and other webif directories to it. That way, you can edit files on your local computer and have the changes be shown in real-time on the router.
- Mount an NFS or SAMBA share somewhere onto your router. (i.e. to /mnt/myshare).
- Copy /www and /usr/lib/webif folders recursively to your network share. (i.e. cp -r /www /mnt/myshare/).
- Remove old /www and /usr/lib/webif folders (i.e. rm -rf /www).
- Symlink the www folder on your network share to the /www folder on your router (i.e. ln -s /mnt/myshare/www /www).
- Symlink the usr/lib/webif folder on your network share to the /usr/lib/webif folder on your router (i.e. ln -s /mnt/myshare/usr/lib/webif /usr/lib/webif).
Afterwards, restart the httpd or reboot your router. This change will persist, so from now on you can work on webif pages by simply editing them on the network share. Changes are shown in real-time as you access the webif on the router.
Sometimes (rarely) after editing pages you may need to restart the httpd or remount the network share to force it to clear all caches.
Functions
A quick grep of the .sh files gives an idea of the functions available:
| Script Name | Function | Short Description | Example |
|---|---|---|---|
| functions.sh | empty() | Determines if a variable is empty, was originally used because if statements were slow in busybox but nbd has since fixed busybox. | |
| functions.sh | equal() | Determines if 2 variables are equal, was originally used because if statements were slow in busybox but nbd has since fixed busybox. | |
| functions.sh | neq() | Determines if 2 variables are not equal, was originally used because if statements were slow in busybox but nbd has since fixed busybox. | |
| functions.sh | exists() | Determines if a file exists, was originally used because if statements were slow in busybox but nbd has since fixed busybox. | |
| functions.sh | is_bcm947xx() | Determines if the router cpu is a Broadcom 947xx | |
| functions.sh | remove_lines_from_file() | doc | |
| functions.sh | load_settings() | Loads settings out of the temporary files Syntax: load_settings filename (Whiterussian Only) | |
| functions.sh | validate() | doc | |
| functions.sh | save_setting() | Used to save nvram variables. The filename is used for the name of the temporary file used to hold the settings until they are applied. The files are stored in /tmp/.webif/ Syntax: save_setting filename nvram-variable "setting to save" | |
| functions.sh | is_package_installed() | This function will only check to see if one package is installed. It will return 0 if the package is installed. | is_package_installed olsrd |
| functions.sh | install_package() | Installs a package, will try to run a ipkg update if it is unable to install the first time and then try again. | install_package olsrd |
| functions.sh | remove_package() | Removes a package | remove_package olsrd |
| functions.sh | update_package_list() | runs ipkg update to update the local repo information | |
| functions.sh | add_package_source() | adds a line to /etc/ipkg.conf with the package information | |
| webif.sh | categories() | doc | |
| webif.sh | subcategories() | doc | |
| webif.sh | ShowWIPWarning() | Placed immediately under header on pages that are a Work In Progress. | |
| webif.sh | ShowUntestedWarning() | Placed immediately under header on pages that are are untested. | |
| webif.sh | has_pkgs() | Used to check to see if a package is installed. If the package is not installed it will display a install message on the page, if the page is installed it will return nothing. It will except and number of packages by seperating them with a space. | haspkgs olsrd chillispot |
| webif.sh | mini_header() | doc | |
| webif.sh | header() | doc | |
| webif.sh | footer() | doc | |
| webif.sh | apply_passwd() | doc | |
| webif.sh | display_form() | doc | |
| webif.sh | list_remove() | doc | |
| webif.sh | handle_list() | doc |
Forms
form.awk gives you predefinded forms to use in you webif page. Most of these are used like formname|input
Normal parameters:
- $1 = type
- $2 = form variable name
- $3 = form variable value
- $4 = (radio button) value of button
- $5 = string to append
- $6 = additional attributes
The current forms are as listed:
| Form Name | Short Description | Syntax |
|---|---|---|
| start_form | doc | |
| onchange | doc | |
| onclick | doc | |
| onclick | doc | |
| button | doc | |
| checkbox | doc | |
| radio | doc | |
| select | Drop down menu use the option form directly below it to add options to it. menu_variable is the name of the menu, $variable is used to load which variable is currently from the options in the menu. You must set $variable somewhere in the file previous to it being used. | menu_variable|$variable |
| option | Options for the drop down menu | option_variable|Displayed Text |
| txtfile | doc | |
| listedit | doc | |
| caption | doc | |
| string | doc | |
| textarea | doc | |
| progressbar | doc | |
| password | doc | |
| upload | doc | |
| submit | doc | |
| helpitem | doc | |
| helptext | doc | |
| helplink | doc | |
| end_form | doc |
Configuration storage
NG-style UCI config vs. nvram
OpenWrt is migrating away from nvram, with it completely removed from buildroot-ng. The webif is doing the same. There are new config functions able to load and store files in the UCI config file format.
Using NVRAM config functions
These functions load and store nvram variables (untyped tuples). An example invocation of saving an nvram varaible is: 'save_setting GROUPNAME VARIABLE=VALUE'.
Using UCI config functions
UCI is the standard configuration file format for OpenWrt packages. It defines a standard file format that can be read with a standard set of code, there-by making all chores of reading and writing configuration files much easier. It is new in OpenWrt Kamikaze, so has not yet been widely adopted by packages. However, the core system configuration already utilizes it.
Example of a UCI configuration file
Configuration files are usually stored in /etc/config/, though they can technically be stored elsewhere. Here is what /etc/config/webif might look like, supplying the configuration for the webif package:
config webif general option firmware_name "" option firmware_version "" option firmware_subtitle "With X-Wrt Extensions" option device_name "" option use_apply_progressbar "1" config webif syslog option ipaddr "" option port "" option size "" option type "" option mark "" option file ""
config webif theme option id "xwrt" config webif cron option enable "1" config webif qos option show_advanced_rules "0" config webif misc option opendns "0"
Using the UCI shell function interface
The UCI config file manipulation functions are in /lib/config/uci.sh. These offer the ability to parse UCI configuration files into shell variables and save and commit new changes. There is automatic support for pending changes to a UCI configuration file too, meaning that saved but not yet committed UCI options will automatically be loaded along with the configuration file itself.
Example displaying and changing a UCI option, example is webif package:
uci_load "webif" # load the configuration echo "somesection.somegroup = $CONFIG_somesection_someoption" # display the current value of the option uci_set "webif" "somegroup" "someoption" "newvalue" # finally, set a new value to the option
Later, when you are ready to commit the changes permanently you would invoke:
uci_commit "webif"
Note that for webif pages you do NOT need to ever call uci_commit since that is handled automatically by the webif system.
| uci_load | Load a UCI configuration file and any pending changes into shell variables. |
| uci_set | Save a value to an option (making it a pending change). |
| uci_commit | Commit all pending changes from uci_save. |
| uci_add | Add a new group or option. |
| uci_remove | Remove a group or option. |
| uci_rename | Rename a group or option. |
Shell variable mapping
UCI options are mapped to shell variables in the following form:
CONFIG_[group]_[option] = Value of an option. CONFIG_[group]_TYPE = Type of the group.
Themes
CSS Theme Rules
We now support multiple CSS themes in the webif. Contributors of new themes should adhere to these rules:
1. The CSS theme must adhere to the existing class/id structure.
2. Changes to class/id names or addition of new ones should be done only if there are no other options, and requires approval of the group.
3. The class/id structure we use should be robust enough to handle various themes.
4. In short, your CSS should adhere to the webif, not the other way around.
5. The CSS theme must support the color switcher. We can have seperate color CSSes for each theme, but it must support all 6 colors.
6. The CSS theme must work in IE 6, IE 7, Opera, and Firefox. You must test it in each.
7. It will not be considered at all for the default theme if it does not work in all browsers. It will be your responsibility to fix bugs and maintain the CSS.
How to create a new CSS theme
CSS themes exist in dedicated subdirectories of /www/themes.
- create a subdirectory named after your theme.
- Copy all CSS files from an existing theme into your new directory. Then, start modifying the CSS files.
- When you want to submit your theme as an official webif extension, submit it to us.
(Ported from OpenWrt's ProgrammersGuideToWebif)
