The server for the BSim Database is distinct from the traditional Ghidra server, although for many use cases it is convenient to have both running and view the BSim server as a loosely coupled extension to the base Ghidra Server. In terms of start-up, shutdown, and configuration however, the two servers are completely separate.
There are two choices for deploying a shared server for the BSim Database: PostgreSQL or Elasticsearch. Both options support multiple users and allow multiple simultaneous connections to the remote server. A single database can ingest large datasets, while still maintaining short query times.
Alternately, a single user can create a BSim database on their local file-system, without a server, by utilizing the H2 database engine integrated into Ghidra. This option is intended for querying against small datasets and does not require installation of additional server software.
A PostgreSQL server, which must be built with a BSim specific extension, runs on a single host and makes efficient use of whatever CPU, memory, and disk resources are made available to it. PostgreSQL is a robust and capable server that should perform well on minimally configured workstations up to high-end production hardware. Source for the BSim extension to PostgreSQL is included as part of the Ghidra installation, but the PostgreSQL source may need to be obtained separately by the database administrator. See “Building the Server”
An Elasticsearch server, which must have a BSim specific plug-in installed, runs as a scalable database that automatically distributes itself across machines in a cluster, allowing individual database queries and requests to be serviced in parallel. The Elasticsearch BSim plug-in is included with the Ghidra installation, but the core server software must be obtained separately by the database administrator. See “Installing the Plug-in”
BSim clients included in the base Ghidra distribution can interface to any of these databases. Users that just want to connect to an existing shared server via a BSim client do not need to install any server software themselves.
In order to use PostgreSQL as a BSim server, it must be built with a BSim specific
extension, provided as part of the Ghidra installation. Prebuilt servers, like those
provided as OS distribution packages, will not work as is with BSim. For users on Linux
and macOS, the Ghidra installation provides a script, make-postgres.sh,
in the module directory Ghidra/Features/BSim/support that builds both the PostgreSQL
server and the BSim extension from source and prepares the installation for use with
Ghidra. If not already included in the Ghidra installation, the source distribution
file, currently postgresql-15.18.tar.gz, can be obtained from the PostgreSQL
website at
https://www.postgresql.org/ftp/source/v15.18
|
The steps to build the PostgreSQL server with the BSim extension then are:
1) If not already present, place the PostgreSQL source distribution file
postgresql-15.18.tar.gz in the Ghidra installation at
$(ROOT)/Ghidra/Features/BSim/support/postgresql-15.18.tar.gz
|
2) From the command-line, within the same directory, run the script make-postgres.sh
cd $(ROOT)/Ghidra/Features/BSim/support
|
./make-postgres.sh
|
Additional packages or software may need to be installed on the host OS in order for the build to complete successfully, OpenSSL in particular is required for BSim. For the full list of PostgreSQL software dependencies, refer to:
https://www.postgresql.org/docs/15/install-requirements.html
|
Once the build has completed successfully, the bsim_ctl command-line script is ready to use for starting a server (see “Starting and Stopping the Server”). The PostgreSQL server software will run out of the Ghidra installation at
$(ROOT)/Ghidra/Features/BSim/build/os/linux_x86_64/postgresql
OR |
$(ROOT)/Ghidra/Features/BSim/build/os/linux_arm_64/postgresql
OR |
$(ROOT)/Ghidra/Features/BSim/build/os/mac_x86_64/postgresql
OR |
$(ROOT)/Ghidra/Features/BSim/build/os/mac_arm_64/postgresql |
Other than having the extension itself, a BSim enabled PostgreSQL server is completely standard, and can be configured like any other stand-alone PostgreSQL server. There are no direct restrictions on modifying the configuration values. A default configuration is provided with this installation that has been tuned specifically for the BSim Database application, so in practice there may be little reason to modify it. But there are a few standard configuration values for the server that might need adjusting. See “Additional Configuration”.
A BSim PostgreSQL server is prepared in two steps: a one-time init that creates and configures the data directory, followed by start to run the server. All operations take an absolute path to the chosen data directory. Configuration is performed entirely through the bsim_ctl init and configure commands; the PostgreSQL configuration files are managed on your behalf.
To initialize a new server, run init once against an empty (or not-yet-existing) data directory. This creates the data directory, tunes the configuration, installs the server certificate, and enables the BSim extension, leaving the server stopped:
$(ROOT)/support/bsim_ctl init /path/to/datadir |
The data directory is created owned by the operating-system account that will run the server. When init is run as an ordinary user, that user becomes the owner. When run as root (via sudo), the --os-user option is required and names a dedicated, non-root service account that will own the data directory and run the server. All subsequent bsim_ctl commands for a data directory must be run either by that owner or by root.
By default (no --keystore) the server is secured with a self-signed certificate and binds to the loopback interface only, which is appropriate for local single-user use. To accept remote client connections, supply a CA-signed server certificate in a password-protected keystore with the --keystore option (see “Security and Authentication”).
WARNING! Deploying the PostgreSQL server with a self-signed certificate (that is, without the --keystore option) requires every client to waive server authentication for loopback connections, since no client trusts such a certificate. That waiver is only appropriate where every account on the local machine is trusted. See “Security and Authentication” for the VM property involved and what it gives up.
Once initialized, the server is started, stopped, restarted, and queried with:
$(ROOT)/support/bsim_ctl start /path/to/datadir |
$(ROOT)/support/bsim_ctl stop /path/to/datadir [--force|-f] |
$(ROOT)/support/bsim_ctl restart /path/to/datadir [--force|-f] |
$(ROOT)/support/bsim_ctl status /path/to/datadir |
The start command takes no options and performs no configuration; if the data directory has not been initialized it reports an error directing you to run init. To use a non-standard port, specify --port at init (or configure) time; any subsequent reference to the server, in the Ghidra client or with the bsim command, must then specify that port with the postgresql:// URL (see “Ghidra and BSim URLs”).
The --force option to stop and restart disconnects clients immediately, rolling back any in-progress transactions, rather than waiting for connections to finish.
Running as a systemd Service (Linux): the server can be installed as a persistent systemd service so that it starts automatically at boot, running the postgres process as the data directory owner. Installing and removing the service must be done as root (via sudo), and the server must be stopped when the service is installed:
sudo $(ROOT)/support/bsim_ctl install-service /path/to/datadir |
sudo $(ROOT)/support/bsim_ctl uninstall-service /path/to/datadir |
Once a service is installed, the start, stop, and restart commands are delegated to systemctl and must be run as root; status reports the service installation and running state.
BSim makes use of PostgreSQL security mechanisms to enforce privileges and authenticate users. The bsim_ctl command wraps the subset of functionality described here, but other adjustments are possible by connecting directly to the server and issuing SQL commands.
The PostgreSQL server, as configured for BSim, only accepts connections via SSL, so communications in transit are always encrypted regardless of the authentication settings.
The SSL/TLS server certificate is established at init time and is independent of the client authentication method described below. If a password-protected keystore is supplied with the --keystore option, its CA-signed certificate and private key are installed and the server listens on all interfaces, allowing remote clients to authenticate the server. If no keystore is supplied, a self-signed certificate is generated and the server binds to the loopback interface only; remote access is disabled.
IMPORTANT: Connections to a newly established server will fail unless the server certificate issuer is considered a trusted certificate issuer (.e.g., Certificate Authority (CA)). A local server reached via a localhost name or address is authenticated exactly like any other server by default.
When a self-signed certificate is used by a server,
client systems must be configured in one of two ways: 1) add the self-signed server certificate
to their set of trusted certificates, or 2) if server is limited to localhost use only local server
authentication may be waived by setting the following VM property (see
support/launch.properties):
-Dghidra.disable.loopback.server.authentication=true |
Waive it only where every account on the local machine is trusted. A loopback connection is not by itself proof of the server's identity: a rogue local process which binds the port ahead of the intended server would be accepted in its place, together with any credential the client then sends it (the administrative database password among them). A deployment which cannot accept that should supply a --keystore with a CA-signed certificate instead, which requires no waiver. Remote deployments must always supply one.
Where the server presents a CA-signed certificate from a --keystore, a client authenticates it using the trust established for that client: the operating system trust store and the Java default trust store. A certificate issued by an authority those already contain requires no client configuration at all. For an internal authority which they do not contain, the client has three options:
Leaves the defaults intact, so all other authorities continue to be trusted. On Unix this requires root access.
Names an additional file or directory of CA certificates which is added to the OS trust store as it is built, so the defaults are augmented rather than replaced, and no root access is needed. Applies to Unix only.
Names a CA certificates file which is used exclusively — the OS and Java default trust stores are then ignored, for this and every other connection the client makes. Every authority the client relies upon must therefore be present within that file. Use this to deliberately restrict a deployment to its own authorities, not merely to add one.
NOTE: the bsim and bsim_ctl utilities set ghidra.cacerts automatically if the property has not
been specified and a file named cacerts is present within a Ghidra
installation root directory. Placing such a file there therefore restricts these
utilities to the authorities it contains. Both properties are described in
support/launch.properties.
PostgreSQL uses the concept of roles to grant access privileges based on particular users. Generally, a user's role is determined by the username used to establish the connection. For BSim, each user role is granted one of two privilege levels: user, which allows read-only access to the server for normal queries, and admin, which additionally allows database creation, ingest, update, and deletion.
BSim supports three different authentication methods, when connecting as a client or during database ingest and maintenance. This method is established for a server by the init command.
bsim_ctl init /path/to/datadir --auth password
This is currently the default authentication mode if --auth is not specified. Users are authenticated via password. A default password 'changeme' is established when the new user is created. Passwords can be changed by the user from the BSim client, or from the command line with the bsim changepassword command (see “bsim”), or can be reset by an administrator via the resetpassword command.
bsim_ctl init /path/to/datadir --auth pki
--cafile "/path/to/rootcert" --cert "/path/to/admincert.p12"
Users are authenticated by PKI certificates. Upon initialization, the BSim server must be provided (via the --cafile option) a file containing the public keys for the certificate authorities used to issue user certificates. The file consists of the authoritative certificates in unencrypted PEM format concatenated together, and must provide a complete chain of trust for each of those authorities (any intermediate certificate authority must be accompanied by the certificates of its issuers, up to and including a root certificate authority), since the server does not look beyond this file when verifying a certificate a client presents.
The administrative user's own certificate must also be provided, via the global --cert option. It supplies the credential used to connect to the new server, and its Distinguished Name is what gets associated with the administrative user name — so no --dn option is given to init.
BSim users must register their certificate with the Ghidra client using the Edit->Set PKI Certificate... menu option from the Project dialog. The BSim client will automatically submit the certificate to a server that requests it, and the password to unlock it will be requested as needed. This is the same mechanism used to a access a PKI protected Ghidra server, and if a user needs access to both a BSim server and Ghidra server that are PKI protected, the servers should probably be configured with the same certificate authorities so that they will accept the same certificate from the user.
With PKI authentication enabled, at the time a new user role is established with the server, the X.509 Distinguished Name, as bound to the user's certificate, must be associated with the user name via the --dn option. See “Adding Users to the Database”.
A server initialized this way identifies each user by their full Distinguished Name. A Common Name on its own is not a unique identity — it is chosen by whoever issues a certificate, and two authorities within the same --cafile may issue certificates carrying the same Common Name to different people, which would let one of them connect as the other. A server initialized by an Ghidra 12.1.x or older matches only the Common Name and continues to do so; see “Adding Users to the Database”.
bsim_ctl init /path/to/datadir
--auth trust
No authentication is performed and privilege is granted based soley on the user name presented. Masquerading is possible; so use of this authentication method is highly discouraged and intended for testing only. When this mode is configured, the PostgrSQL server will listen on the loopback/localhost interface only.
The authentication method is established when the data directory is initialized with the init command; starting the server does not change it. If the settings really need to be changed, the configure command can be issued. It takes the same options as the init command and must be run while the server is running, so that the administrator can be authenticated and any credential the new authentication method requires can be established before the change takes effect. The server is left running and the change takes effect when it is next restarted.
$(ROOT)/support/bsim_ctl configure
/datadir/path --auth password |
Using the configure command on a server with an established set of users will likely require other disruptive changes to create passwords or associate Distinguished Names with users, if they didn't exist before. The administrator's own credential is handled by configure itself: a change to password prompts for a new password if they have none, and a change to pki requires their certificate (the global --cert option) so their identity mapping can be established from it. Every other user must then be given a password with resetpassword, or a Distinguished Name with adduser, once the server is restarted.
If it is determined that only the database administrators have OS level, local, access to the server's host machine, they can choose to use the noLocalAuth option as part of the init or configure commands. This disables authentication for users connecting to the server by the 'localhost' interface. This may facilitate the use of scripts for ingest etc., where working with passwords is cumbersome. Authentication is still enforced for any remote connection.
NOTE: A host whose local accounts are not all trusted should be given a --keystore so that the configured authentication applies to local connections too.
Only local authentication is affected. The --auth mode is still recorded and continues to determine what adduser establishes for each user — a password, or a certificate identity mapping — so those credentials are already in place if configure is later used with a --keystore to enable remote access. Users should therefore continue to be added in accordance with the configured authentication mode, even though local connections are not currently authenticated.
The PostgreSQL administrative role established when the server is initialized (the --user option of bsim_ctl init, defaulting to the invoking user) becomes the administrator for that server. No other username/role is initially known to the server. New usernames/roles can be added to the server using the following command:
$(ROOT)/support/bsim_ctl adduser /path/to/datadir username |
$(ROOT)/support/bsim_ctl adduser /path/to/datadir username --dn "CN=Firstname User,ST=MD,C=US" |
If password authentication has been set for the server, the new user's password will initially be set to 'changeme'. If PKI authentication has been set for the server, the Distinguished Name, as bound to the new user's certificate, must be provided when issuing the adduser command, via the --dn option. The Distinguished Name must be presented as a string containing a comma separated sequence of attribute/value pairs that uniquely identifies a certificate.
IMPORTANT: the server matches the entire Distinguished Name, so no attribute may be omitted, and it must be given exactly as the server renders the subject of the user's certificate: in RFC 2253 form (most specific attribute first, no spaces around the separating commas), matching case and spacing. No normalization is performed on either side of the comparison, so specifying it correctly is the administrator's responsibility. For a given certificate the required form is reported by:
openssl x509 -noout -subject
-nameopt RFC2253 -in certfile |
The name actually registered is echoed by adduser, and the name registered for every user can be reviewed at any time with the listusers command — which together are the means of confirming a Distinguished Name was entered correctly. A user whose registered name does not match the certificate they present will be unable to connect; re-running adduser for that user replaces it.
NOTE: a server which was initialized by Ghidra 12.1.x or older matches only the Common Name (CN) taken from the Distinguished Name, rather than the whole of it. Such a server keeps doing so, because the entries it already holds for its users were registered that way and cannot be re-derived (each user's certificate is held by that user). adduser follows whichever the server is configured for, so the --dn option is supplied the same way in either case; listusers and status report which is in use.
New users are by default only given user permissions, meaning that they can only place queries to the database and cannot ingest, update, or delete data. The new user can be given admin privileges (by an existing administrator) by issuing the command:
$(ROOT)/support/bsim_ctl changeprivilege username admin |
The relevant configuration files are at the top level of the data directory:
postgresql.conf |
pg_hba.conf |
Within postgresql.conf, BSim appends two clearly
delimited blocks after the standard PostgreSQL defaults:
The BSim-managed settings (SSL/TLS, interface binding, client authentication, and logging). Do not edit this block - it is regenerated every time configure is run, so changes made here are lost. Use bsim_ctl configure to change these settings.
The performance-tuning settings. This block is written once at init and is preserved by configure. To tune an existing server, edit the values in this block and restart the server.
More generally, any setting outside the managed block - the tunable block, or the standard PostgreSQL defaults above it - may be edited and is preserved across a configure; only the managed block is regenerated. The performance parameters provided in the tunable block are:
This controls the amount of RAM available for caching database pages across all connections to the server. The default should be reasonable in most situations, but for large databases or many simultaneous connections it might make sense to increase this.
The maximum memory used by an internal hash table or sort operation before it spills to temporary disk files.
Controls how often the server forces database pages to be written back out to the file-system. The default is set to minimize disk writes when ingesting large numbers of records in one session; there is usually little reason to change it.
The minimum TLS version and cipher suites live in the managed block (currently ssl_min_protocol_version = 'TLSv1.2') and, like the other managed settings, are established by init/configure rather than edited directly.
The pg_hba.conf file is used to configure which
connections the server accepts for a particular outward facing IP address and what
security mechanisms are enforced for those connections. Currently all addresses are
configured to accept SSL connections only, except possibly for 'localhost'.
Administrators can currently filter connections
based on usernames and the particular database (which corresponds to Ghidra's concept
of repository).
With the default --auth trust setting the server accepts all connections from all users. Without a --keystore the server binds to the loopback interface only, so this applies to local connections; supplying a keystore enables remote access, for which a stronger --auth mode should be used.
The installation-wide default configuration values are held in a serverconfig.xml template (in $(ROOT)/Ghidra/Features/BSim/data), analogous to the Ghidra
Server's server.conf. It provides reasonable defaults that
differ from the standard PostgreSQL defaults and is read by init (and by configure when it regenerates the managed block).
Entries marked tunable="true" (the
performance settings) are written to the editable tunable block of a new data directory's
postgresql.conf.
There are therefore two ways to adjust the performance settings:
Edit the values in the tunable block (between # BSIM-TUNABLE-BEGIN and # BSIM-TUNABLE-END) of that directory's postgresql.conf, then restart the server. These edits are
preserved across configure.
Edit the corresponding serverconfig.xml entry
before running init; the new value becomes the installation-wide
default for any data directory subsequently initialized.
Remote client access to the PostgreSQL server may have network firewalls which must be considered and properly configured to allow network access. Both dedicated network firewalls and OS-level firewalls must be considered.
Firewall configurations are beyond the scope of this document, however for simple
single-node installations on Linux the firewall-cmd
may be used to allow incoming connections to the appropriate port (TCP port 5432 is the
default for PostgreSQL). This does not consider network firewall devices which may
also impact connectivity.
sudo firewall-cmd --permanent --add-port=5432/tcp && sudo firewall-cmd --reload |
NOTE: The above Linux firewall command assumes the firewalld
package has been installed on the system.
A full description of how to configure an Elasticsearch cluster is beyond the scope of this document. In particular, the bsim_ctl command-line, as described in “PostgreSQL Configuration”, does not apply to Elasticsearch. Complete documentation for administering a database is available on-line from the Elasticsearch website.
The following discussion describes how to set up a toy, or single node, server, using the
free and open Elasticsearch distribution. This distribution includes a REST API for administering
a database, which can be accessed using curl commands or some other method
to send HTTP requests directly to the node.
In order to make use of Elasticsearch with BSim, the database administrator must install the lsh.zip plug-in as part of the Elasticsearch deployment. The plug-in is available in the Ghidra extension named BSimElasticPlugin. The extension may be unzipped to a temporary location or use Ghidra's Install Extensions capability which will unpack it into the users platform-specific config directory indicated in the detail view of the Install Extensions window. The extension is not used directly by Ghidra and is only needed for the lsh.zip elasticsearch plug-in which must be installed on every node of the cluster before a BSim repository can be created. The description below shows how to enable the BSim plug-in for a single node, but this will need to be repeated for any additional nodes.
Refer to the README.md
file within the unpacked extension for important plugin details. The plugin file
stipulates a specific elasticsearch version and may require an adjustment and repack.
Assuming the add-on has been unpacked, the plug-in can be installed to a single node using the elasticsearch-plugin command in the bin directory of the node's Elasticsearch installation.
bin/elasticsearch-plugin install
file:///path/to/ghidra/extension/BSimElasticPlugin/data/lsh.zip |
Replace the initial portion of the absolute path in the URL to point to the Ghidra installation. Once the plug-in is installed, the toy deployment can be (re)started from the command-line by running
bin/elasticsearch |
This will dump logging messages to the console, and you should see [lsh] listed among the loaded plug-ins as the node starts
up.
The open Elasticsearch distribution starts with password authentication
enabled by default. When a node is started up for the first time, as described above, an
elastic user is created with a randomly
generated password that is reported, once, to the console. For a toy deployment, it may
be convenient to add additional users via curl commands. The
following example creates a user named ghidrauser
with a default password "changeme", using the elastic users credentials.
The generated password for the elastic user must be substituted for the XXXXXX
at the beginning of the command.
curl -k -u elastic:XXXXXX -X POST "https://localhost:9200/_security/user/ghidrauser?pretty" -H 'Content-Type: application/json' -d'
{
"password" : "changeme",
"roles" : [ "viewer" ],
"full_name" : "Ghidra User",
"email" : "ghidrauser@example.com"
}
'
Elasticsearch uses the concept of roles to grant access privileges to particular users. The built-in role viewer, as in the example above, can be used to grant users read-only access to a database. The built-in superuser role grants administrator privileges.
Once a user has been created they may change their own password from the BSim client, or from the command line with the bsim changepassword command (see “bsim”).
Assuming an Elasticsearch cluster is running and the plug-in has been properly installed, all other parts of BSim interact transparently with the cluster. The bsim command, described in Ingesting Executables, and the Ghidra/BSim client, described in Querying a BSim Database, require no additional configuration to work with Elasticsearch, except users must provide the correct URL to establish a connection. Elasticsearch communicates over https, and BSim clients automatically assume they are communicating with Elasticsearch when they see this protocol. Alternatively, the protocol may be specified as elastic when using the bsim command. Elasticsearch use by BSim assumes a default port of 9200 unless otherwise specified when specifying the server host. See “Ghidra and BSim URLs” for additional information about URLs.
Remote client access to the Elasticsearch server/cluster may have network firewalls which must be considered and properly configured to allow network access. Both dedicated network firewalls and OS-level firewalls must be considered.
Firewall configurations are beyond the scope of this document, however for simple
single-node installations on Linux the firewall-cmd
may be used to allow incoming connections to the appropriate port (TCP port 9200 is the
default for elasticsearch). This does not consider network firewall devices which may
also impact connectivity.
sudo firewall-cmd --permanent --add-port=9200/tcp && sudo firewall-cmd --reload |
NOTE: The above Linux firewall command assumes the firewalld
package has been installed on the system.
If using either PostgreSQL or Elasticsearch the server must be properly configured and running before a database can be created. In the case of an H2 file-based database there is no server requirement. Only after a database has been created can data be ingested or queries performed. In this context, a database is a single container of reverse engineered functions. Metadata pertaining to executables and call-graph relationships is also stored, but the principle database record describes a function. A single PostgreSQL or Elasticsearch server can hold multiple independent databases.
A database is created using the bsim command script. The basic command looks like
$(ROOT)/support/bsim createdatabase bsimURL config_template |
A BSim database is completely distinct from the Ghidra Server or Ghidra project, so the executables and functions contained within do not need to coincide at all.
The Ghidra GUI client specifies a BSim database with its explicit characteristics (i.e., DB type, name, host/port if applicable, etc.), while the bsim command accepts a bsimURL which includes similar details (see “Ghidra and BSim URLs” for more details).
The config_template parameter passed to bsim createdatabase names a collection of specific configuration values for the newly created database. A standard Ghidra distribution provides a number of predefined templates (See below) designed for specific database use cases. It is simplest to use a predefined template when creating a database, but it is possible to edit an existing template or create a new template (See “Creating Database Templates”).
There are two critical database properties being determined by the template that need to be kept in mind: the index tuning and the weighting scheme relative to the size of the database. The two pieces of the template name, separated by the '_' character, refer to these concerns.
The index tuning affects the use of the database by trading off between, the time required to perform individual queries, the amount of variation between matching functions a query can tolerate, and the amount of storage required per database record. Ideally, the database is tuned, before the initial ingest occurs, to the anticipated size of the database. The database can trade off storage size (per record) and latency for overall query response time, but the decision needs to be made before the database is populated. Currently there is a medium tuning that is ideal for repositories that will store on the order of 10 million functions. There is also a large tuning, which uses more storage but can maintain fast query times for databases with 100 million functions or more. There is a large overlap for these tunings, so if its unclear how large a database might grow, go ahead and use the medium tuning.
The weighting scheme affects how BSim views the relative importance of individual code constructs within a function. Code constructions are extracted as features, and each feature is assigned a weight. The basic schemes are: 32 for 32-bit compiled code, 64 for 64-bit code. The scheme that matches the predominant form of code in the repository being ingested should be used. Mixed schemes are possible, but a corpus which is predominantly 32-bit, even with a small number of 64-bit executables mixed in, should still use the 32-bit weights.
There are some weighting schemes designed for more specialized code. The 64_32 scheme is for 64-bit code using 32-bit pointers. The nosize scheme allows better matching of 32-bit functions to 64-bit functions, when they are compiled from the same source. The cpool scheme is designed for Java byte-code or Dalvik executables. For more discussion of weighting, see “Weighting Software Features”.
The full template name incorporates both an index tuning and a weight scheme. Some common examples of template names:
A medium index tuning with a weighting scheme designed for 32-bit executables.
A medium index tuning with a weighting scheme designed for 64-bit executables.
A 32-bit weighting scheme with tuning for a large database size.
A medium index tuning with a weighting scheme for Java executables.
A medium index tuning with a weighting scheme allowing matches between 32-bit and 64-bit code.
There is some facility to tailor a specific BSim database instance so that it can ingest and/or report information about executables or functions to make results more useful for a specific project or user. Capabilities can be added after a database has been created and is running by issuing specific bsim commands, but they can also be added to a configuration template prior to creating the database, which provides a record of the specific additions should the database instance need to be recreated or multiple tailored instances be deployed. For additions that allow the ingest of more metadata about executables or functions, users must provide additional scripts to Ghidra during the ingest process in order to read in or discover the new metadata.
The Name, Owner, and Description associated with a BSim instance can be trivially tailored with the bsim setmetadata command.
$(ROOT)/support/bsim setmetadata bsimURL --name "BSim Database" |
$(ROOT)/support/bsim setmetadata bsimURL --owner "Administrators" |
$(ROOT)/support/bsim setmetadata bsimURL --description "Files of interest" |
This information is displayed in various windows by the BSim client. The values can be changed at any time and do not otherwise affect the records contained in the database. Multiple command-line parameters can be fed to bsim setmetadata so long as each one starts with --name, --owner, or --description respectively. Quoting of values may be necessary to get some strings to be interpreted as a single command-line parameter.
BSim provides the powerful ability to associate new types of metadata with each executable that the database ingests. Any method of categorizing executables that describes an executable with a simple string value, referred to here as an executable category, can be added as a field to the database. With only minor adjustments to the ingest process, new category values can be automatically attached to incoming executables and are treated like any other executable field that BSim understands. Category values are retrieved with queries, can be used for filtering, and show up as sortable columns in result tables.
All categories have a formal name (or type), which is used both in the ingest process (See below) and as the label for table columns. The name can contain alphanumeric characters or punctuation from the limited set, ' ._:/()'. For each executable there can be zero, one, or more string values associated with the category. No value is required for the executable, and any single value can be used for filtering (either the executable is labeled with the value or it is not) even if there are multiple values for that category. If there are multiple values, a query that matches the executable will return all the values as a single sorted column entry.
It is also possible to create a special time-based category. This category can have any name as above, but instead of associating string values with the executable, it associates a single time-stamp. The time-stamp has precision down to the millisecond and provides filtering and sorting based on time. Internally, this new category repurposes the column storage originally providing an executable's Ingest Date field. As a result, any BSim instance can have only one time category and only one time-stamp within it. The ingest scripting must provide any actual time-stamp value for the executable, or the database will fill in the "epoch", 12:00 am, Jan 1, 1970.
A new category can be added to the database at any time using the bsim addexecategory command.
$(ROOT)/support/bsim addexecategory bsimURL MyCategoryName |
$(ROOT)/support/bsim addexecategory bsimURL MyTimeField date |
The single time-stamp field can be renamed by appending the keyword "date" to the command after the name of the category. Once a category is added, the corresponding program options set for any new executables will automatically read into the database as part of the ingest process. Previously ingested executables, assuming they have the new program options set, can be updated within the BSim database using one of the bsim generateupdates command variants. In either case, the relevant program options typically need to be filled by running a Ghidra script (See “Ingesting Executable Categories”). There is currently no method for deleting a category once it has been created.
BSim can be configured to recognize specific Function Tags, which are named Boolean properties on individual functions within an executable. Within a Ghidra program, any number of different function tags can be established by the user and are used to label individual functions or specific subsets of functions that share a particular property. This would typically be used to label classes of functions that are important to analysts, unpacked functions could be labeled with the tag UNPACKED for instance.
In order for BSim to recognize specific function tags, they must be individually registered with the BSim database. These tags are then automatically ingested into the database, along with the other standard metadata describing functions, and can be used to filter match results when querying the database. A function tag has a formal name, which can be displayed as part of the function header within the main code browser and is used for BSim columns and filter labels. Once the tag is created for a program, functions universally have the tag as a Boolean property, either the name applies to a function or it doesn't, and arbitrary subsets can be tagged with that name.
A tag must be registered with a BSim database before it can be used as a filter or seen in results. A tag can be registered at any time with the bsim addfunctiontag command.
$(ROOT)/support/bsim addfunctiontag bsimURL MyTagName |
The new tag will automatically be read in when any new executables are ingested. If previously ingested executables already had the new tags before they were registered, their metadata within BSim database can be updated using the bsim generateupdates command variants. BSim is limited to 29 registered tag names, and there is currently no way to remove a tag once it has been registered.
It is possible to create tailored database configuration templates so that implementers have a permanent and accessible record of a particular set-up and don't need to repeatedly issue bsim setmetadata and bsim addexecategory when creating a database. Other aspects of a database can also be manipulated, like weighting schemes and index tuning, but doing this properly is beyond the scope of this document. A database template is the basic set of configuration parameters used to set up BSim database instance. The configuration parameters are established for a particular database when the bsim createdatabase command is run (See “Creating a Database”). The template name passed on the command-line actually identifies an XML file-name, appended with the '.xml' suffix, in the directory:
$(ROOT)/Ghidra/Features/BSim/data |
The file has a root tag of <dbconfig>, and the first child tag of this root is the <info> tag. This tag contains all the metadata tags that can be easily changed or added to the database. A list of the metadata tags follows. The metadata is provided as formal text content within the tag, and none of the tags currently take attributes.
| XML Tag | Description |
<name> |
Name of the database |
<owner> |
Owner of the database |
<description> |
An overarching description of the database |
<major> |
Major decompiler version used for ingest (Should be set to zero) |
<minor> |
Minor decompiler version used for ingest (Should be set to zero) |
<settings> |
Specific settings for the signature strategy (Should be set to zero) |
<execategory> |
The name of an executable category (to be) defined for this database instance |
<datename> |
The name of the timestamp column |
<functiontag> |
The name of a function tag (to be) registered with this database instance |
There can be multiple <execategory> tags if more than one category is desired and both <execategory> and <datename> are optional tags. The date column name defaults to 'Ingest Date' and is drawn from the corresponding Ghidra program option. The tag order needs to be preserved. There can be multiple <functiontag> tags, one for each function tag to be registered with the database.
It is easiest to copy an existing template and just edit the tags described above. The remaining tags in the file are more dangerous to manipulate. The <k> and <L> tags pertain to the index tuning. The <weightsfile> tag gives the name of the weights file, within the same directory, which is also another XML file. It is simplest to choose from the existing weight files provided with the distribution. See “Weighting Software Features”.