<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.3.2">Jekyll</generator><link href="/feed" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-08-08T15:12:47-06:00</updated><id>/feed</id><title type="html">A bit about MailStore</title><entry><title type="html">Recreate Recovery Records from FDB file</title><link href="/mailstoreserver,/mailstorehome/Recreate-Recovery-Records-from-FDB-file/" rel="alternate" type="text/html" title="Recreate Recovery Records from FDB file" /><published>2024-08-08T00:00:00-06:00</published><updated>2024-08-08T00:00:00-06:00</updated><id>/mailstoreserver,/mailstorehome/Recreate%20Recovery%20Records%20from%20FDB%20file</id><content type="html" xml:base="/mailstoreserver,/mailstorehome/Recreate-Recovery-Records-from-FDB-file/"><![CDATA[<p>Recovery records can be recreated from the <code class="language-plaintext highlighter-rouge">MailStoreFileGroup.fdb</code> file using the <code class="language-plaintext highlighter-rouge">RecreateRecoveryRecords</code> command.
<!--more-->
Recovery Records do not need to be rebuilt in most cases unless there is underlying damage to the database and the <em>Check Integrity</em> feature reports problems with the recovery records.</p>

<p>You must resolve all underlying issues with the database before rebuilding recovery records. Any issues relating to the <em>FDB</em> file, missing messages, or Content <em>DAT</em> files must be resolved before rebuilding recovery records.</p>

<h2 id="mailstore-server">MailStore Server</h2>

<ol>
  <li>Login to <em>MailStore Client</em> as admin.</li>
  <li>Open <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Note the ID number for the damaged Storage Location.</li>
  <li>Right-click the Storage Location and set the state to <em>Disabled</em>.</li>
  <li>
    <p>Go to <em>Administrative Tools</em> &gt; <em>API</em> &gt; <em>Command Line</em> and use the <a href="https://help.mailstore.com/en/server/Administration_API_-_Function_Reference#RecreateRecoveryRecords">RecreateRecoveryRecords</a> command to rebuild the Storage Location ID as noted in the previous step.</p>

    <pre><code class="language-mailstoreAPI"> RecreateRecoveryRecords --id=1
</code></pre>
  </li>
  <li>Return to <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Right-click the Storage Location and set the state to <em>Normal</em>.</li>
  <li>Perform a <em>Check Integrity</em> to confirm the state of the database.</li>
  <li>If desired, set the <em>Storage Location</em> to <em>Archive Here</em> status.</li>
</ol>

<h2 id="mailstore-service-provider-edition">MailStore Service Provider Edition</h2>

<p>See the <em>Recreate Recovery Records</em> command in the <a href="https://help.mailstore.com/en/spe/Instance_Management">Instance Management</a> documentation.</p>

<h2 id="mailstore-home">MailStore Home</h2>

<ol>
  <li>Close MailStore Home.</li>
  <li>Find your MailStore Home data directory. This may be under <em>%userprofile%</em> in a normal install, or alongside the <em>MailStoreHome.exe</em> file using a portable installation.</li>
  <li>Open a command prompt in the MailStore Home data directory.</li>
  <li>Run the following command: <code class="language-plaintext highlighter-rouge">MailStoreHome.exe /recreaterecoveryrecords</code>.</li>
  <li>Open MailStore Home.</li>
  <li>Perform a <em>Check Integrity</em> to confirm the state of the database.</li>
</ol>]]></content><author><name></name></author><category term="MailStoreServer," /><category term="MailStoreHome" /><category term="data recovery" /><summary type="html"><![CDATA[Recovery records can be recreated from the MailStoreFileGroup.fdb file using the RecreateRecoveryRecords command.]]></summary></entry><entry><title type="html">Recreate FDB file from Recovery Records</title><link href="/mailstoreserver,/mailstorehome/Recreate-FDB-file-from-Recovery-Records/" rel="alternate" type="text/html" title="Recreate FDB file from Recovery Records" /><published>2024-08-07T00:00:00-06:00</published><updated>2024-08-07T00:00:00-06:00</updated><id>/mailstoreserver,/mailstorehome/Recreate%20FDB%20file%20from%20Recovery%20Records</id><content type="html" xml:base="/mailstoreserver,/mailstorehome/Recreate-FDB-file-from-Recovery-Records/"><![CDATA[<p><code class="language-plaintext highlighter-rouge">MailStoreFileGroup.fdb</code> can be rebuilt in-place from the recovery records.
<!--more-->
Please ensure you have a backup of the Storage Location before proceeding.</p>

<p>The <em>MailStoreFileGroup.FDB</em> file should not need to be repaired or recreated unless there is a problem with the file itself. As the <em>FDB</em> file is open at all times while MailStore is running, it is possible that the file can be missed by third-party backup software, or otherwise become damaged.</p>

<p>Do not place MailStore’s database files in a directory monitored by a real-time synchronization product such as Dropbox, OneDrive, or Google Drive. These products are not designed to handle the constant changes to the database files and will result in an inconsistent database.</p>

<h2 id="mailstore-server">MailStore Server</h2>

<p>Disable the damaged Storage Location:</p>

<ol>
  <li>Login to <em>MailStore Client</em> as admin.</li>
  <li>Open <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Note the ID number for the damaged Storage Location, use this ID in the <code class="language-plaintext highlighter-rouge">--id=1</code> parameter in the following steps.</li>
  <li>Go to <em>Administrative Tools</em> &gt; <em>Management API</em> &gt; <em>Command Prompt</em>.</li>
  <li>
    <p>Enter the following command:</p>

    <pre><code class="language-mailstoreAPI"> SetStoreRequestedState --id=1 -requestedState=disabled
</code></pre>
  </li>
</ol>

<p>Rebuild the <em>FDB</em> file:</p>

<ol>
  <li>Remain in <em>Administrative Tools</em> &gt; <em>Management API</em> &gt; <em>Command Prompt</em>.</li>
  <li>
    <p>Enter the following command:</p>

    <ul>
      <li><code class="language-plaintext highlighter-rouge">RecoverStore --id=1</code></li>
    </ul>
  </li>
</ol>

<p>Re-enable and verify the Storage Location:</p>

<ol>
  <li>Open <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Right-click on the Storage Location and select <em>Check Integrity</em>.</li>
  <li>If the integrity check is successful, right-click on the Storage Location and set it to <em>Normal</em> or <em>Archive Here</em>.</li>
</ol>

<h2 id="mailstore-service-provider-edition">MailStore Service Provider Edition</h2>

<p>Please contact support for assistance.</p>

<h2 id="mailstore-home">MailStore Home</h2>

<p>Please post in the <a href="https://community.mailstore.com/">MailStore Home Community</a> for assistance.</p>]]></content><author><name></name></author><category term="MailStoreServer," /><category term="MailStoreHome" /><category term="data recovery" /><summary type="html"><![CDATA[MailStoreFileGroup.fdb can be rebuilt in-place from the recovery records.]]></summary></entry><entry><title type="html">Rearchiving journal messages into separate archives</title><link href="/mailstoreserver/mailstorespe/Rearchiving-journal-messages-into-separate-archives/" rel="alternate" type="text/html" title="Rearchiving journal messages into separate archives" /><published>2023-12-22T00:00:00-07:00</published><updated>2023-12-22T00:00:00-07:00</updated><id>/mailstoreserver/mailstorespe/Rearchiving%20journal%20messages%20into%20separate%20archives</id><content type="html" xml:base="/mailstoreserver/mailstorespe/Rearchiving-journal-messages-into-separate-archives/"><![CDATA[<p>In the case where an Exchange Journal mailbox was archived into a single archive, you may wish to re-archive the messages into the correct archives.</p>

<p>This process will work both for messages in a journal format, and when messages for multiple users have been mixed into one archive.</p>

<h2 id="planning-and-preparation">Planning and preparation</h2>

<h3 id="check-temporary-disk-space"><strong>Check Temporary Disk Space</strong></h3>

<ul>
  <li>Go to <em>Administrative Tools</em> &gt; <em>Users and Archives</em> &gt; <em>Archives</em>.</li>
  <li>Enable <em>Show Statistics (slow)</em>.</li>
  <li>Note the <em>Total Size MB</em> of the existing journal mailbox.</li>
  <li>Consider that additional free space will be needed to leave the old messages on disk until the new archive is complete.</li>
</ul>

<h3 id="consider-how-you-will-delete-the-existing-messages"><strong>Consider how you will delete the existing messages</strong></h3>

<ul>
  <li>If deleting entire MailStore archive, create a new <em>Storage Location</em>, then <em>Detach</em> &amp; <em>Delete</em> the current one.</li>
  <li>If you have other data you want to retain then you will delete the journal archive, and finally compact Storage Locations to release disk space back to the OS.</li>
</ul>

<h3 id="unblock-admin-access"><strong>Unblock Admin Access</strong></h3>

<ul>
  <li>
    <p>Navigate to <em>Administrative Tools</em> &gt; <em>Compliance</em> &gt; <em>Compliance General</em>.</p>
  </li>
  <li>
    <p>Unblock admin access to the archive.</p>
  </li>
</ul>

<h2 id="steps-to-organize-exchange-journal-messages">Steps to Organize Exchange Journal Messages</h2>

<h3 id="1-create-an-export-e-mail-profile">1. <strong>Create an ‘Export E-mail’ Profile</strong></h3>

<ul>
  <li>
    <p>Access the <em>Export E-mail</em> panel.</p>
  </li>
  <li>Create new profile: <em>E-mail Files</em> &gt; <em>Directory (File System)</em>.</li>
  <li>Set <em>Backup/Export Scope</em> to <em>EML Files</em>.</li>
  <li>Select <em>Journal</em> mailbox.</li>
  <li>Enter path with available space (e.g., <em>C:\EML</em>).</li>
  <li>Ensure <em>Retain folder structure</em> is checked.</li>
  <li>Enable <em>Update existing export</em>.</li>
  <li>Leave <em>Run after clicking Finish</em> checked and click <em>Finish</em>.</li>
</ul>

<h3 id="2-optional-create-a-new-storage-location">2. <strong>(Optional) Create a New Storage Location</strong></h3>

<ul>
  <li>Open <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Create a new Storage Location.</li>
  <li>Set the new Storage Location to <em>Archive Here</em> status.</li>
  <li>(Optional) Detach current Storage Location (right-click &gt; <em>Detach</em>).</li>
</ul>

<h3 id="3-optional-remove-messages-from-the-archive">3. (Optional) Remove messages from the archive</h3>

<ul>
  <li>Consider performing the steps in the <em>Clean-up the existing archive</em> section of this article at this time if you have limited disk space.</li>
</ul>

<h3 id="3-create-an-archive-e-mail-profile">3. <strong>Create an ‘Archive E-mail’ Profile</strong></h3>

<ul>
  <li>
    <p>Go to <em>Archive E-mail</em> panel.</p>
  </li>
  <li>Create new profile: <em>E-mail Files</em> &gt; <em>EML and MSG files</em>.</li>
  <li>Choose <em>Multiple Users</em>.</li>
  <li>Set <em>Directory</em> to path from export profile (e.g., <em>C:\EML</em>).</li>
  <li>Configure Target Folders:
    <ul>
      <li>Received: (User /) <em>Journal Incoming</em>.</li>
      <li>Sent: (User /) <em>Journal Outgoing</em>.</li>
    </ul>
  </li>
  <li>For unknown email addresses, select <em>Don’t archive message with unknown e-mail addresses</em>.</li>
  <li>Leave <em>Delete them in origin mailbox</em> unchecked.</li>
  <li>Leave <em>Run after clicking Finish</em> checked and click <em>Finish</em>.</li>
</ul>

<h3 id="4-check-for-missingskipped-users">4. <strong>Check for Missing/Skipped Users</strong></h3>

<ul>
  <li>
    <p>In <em>Progress View</em>, click <em>Details</em>.</p>
  </li>
  <li>Open <em>Debug Log</em> in Notepad.</li>
  <li>Note email addresses with <em>ServerUnmappableException</em>.</li>
  <li>Create user accounts for missing users in <em>Administrative Tools</em> &gt; <em>Users and Archives</em> &gt; Users.</li>
  <li>Re-run <em>Archive E-mail</em> profile.</li>
  <li>Delete user accounts upon completion.</li>
</ul>

<h2 id="optional-clean-up-the-existing-archive">(Optional) Clean-up the existing archive</h2>

<h2 id="detach-storage-location-and-delete-the-files"><strong>Detach Storage Location and delete the files</strong></h2>

<ul>
  <li>Go to <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Right-click on your old Storage Location, select <em>Detach</em>.</li>
  <li>Use Windows Explorer to delete the files on disk.</li>
  <li>If you use MailStore’s internal backup feature, check <em>restore.csv</em> to determine which files belong to the old Storage Location and delete them from the backup target as well.</li>
</ul>

<h3 id="delete-the-journal-mailbox"><strong>Delete the Journal Mailbox</strong></h3>

<ul>
  <li>Navigate to <em>Administrative Tools</em> &gt; <em>Users and Archives</em> &gt; Users.</li>
  <li>Delete <em>Journal</em> mailbox.</li>
  <li>Go to <em>Administrative Tools</em> &gt; <em>Users and Archives</em> &gt; Archives.</li>
  <li>Delete <em>Journal</em> archive.</li>
  <li>Compact first Storage Location in <em>Administrative Tools</em> &gt; <em>Storage</em> &gt; <em>Storage Locations</em>.</li>
  <li>Repeat compaction for additional Storage Locations.</li>
</ul>]]></content><author><name></name></author><category term="MailStoreServer" /><category term="MailStoreSPE" /><category term="Exchange" /><category term="data management" /><summary type="html"><![CDATA[In the case where an Exchange Journal mailbox was archived into a single archive, you may wish to re-archive the messages into the correct archives. This process will work both for messages in a journal format, and when messages for multiple users have been mixed into one archive. Planning and preparation Check Temporary Disk Space Go to Administrative Tools &gt; Users and Archives &gt; Archives. Enable Show Statistics (slow). Note the Total Size MB of the existing journal mailbox. Consider that additional free space will be needed to leave the old messages on disk until the new archive is complete. Consider how you will delete the existing messages If deleting entire MailStore archive, create a new Storage Location, then Detach &amp; Delete the current one. If you have other data you want to retain then you will delete the journal archive, and finally compact Storage Locations to release disk space back to the OS. Unblock Admin Access Navigate to Administrative Tools &gt; Compliance &gt; Compliance General. Unblock admin access to the archive. Steps to Organize Exchange Journal Messages 1. Create an ‘Export E-mail’ Profile Access the Export E-mail panel. Create new profile: E-mail Files &gt; Directory (File System). Set Backup/Export Scope to EML Files. Select Journal mailbox. Enter path with available space (e.g., C:\EML). Ensure Retain folder structure is checked. Enable Update existing export. Leave Run after clicking Finish checked and click Finish. 2. (Optional) Create a New Storage Location Open Administrative Tools &gt; Storage &gt; Storage Locations. Create a new Storage Location. Set the new Storage Location to Archive Here status. (Optional) Detach current Storage Location (right-click &gt; Detach). 3. (Optional) Remove messages from the archive Consider performing the steps in the Clean-up the existing archive section of this article at this time if you have limited disk space. 3. Create an ‘Archive E-mail’ Profile Go to Archive E-mail panel. Create new profile: E-mail Files &gt; EML and MSG files. Choose Multiple Users. Set Directory to path from export profile (e.g., C:\EML). Configure Target Folders: Received: (User /) Journal Incoming. Sent: (User /) Journal Outgoing. For unknown email addresses, select Don’t archive message with unknown e-mail addresses. Leave Delete them in origin mailbox unchecked. Leave Run after clicking Finish checked and click Finish. 4. Check for Missing/Skipped Users In Progress View, click Details. Open Debug Log in Notepad. Note email addresses with ServerUnmappableException. Create user accounts for missing users in Administrative Tools &gt; Users and Archives &gt; Users. Re-run Archive E-mail profile. Delete user accounts upon completion. (Optional) Clean-up the existing archive Detach Storage Location and delete the files Go to Administrative Tools &gt; Storage &gt; Storage Locations. Right-click on your old Storage Location, select Detach. Use Windows Explorer to delete the files on disk. If you use MailStore’s internal backup feature, check restore.csv to determine which files belong to the old Storage Location and delete them from the backup target as well. Delete the Journal Mailbox Navigate to Administrative Tools &gt; Users and Archives &gt; Users. Delete Journal mailbox. Go to Administrative Tools &gt; Users and Archives &gt; Archives. Delete Journal archive. Compact first Storage Location in Administrative Tools &gt; Storage &gt; Storage Locations. Repeat compaction for additional Storage Locations.]]></summary></entry><entry><title type="html">MailStore Gateway Redundancy - Active-Active configuration</title><link href="/mailstoregateway/MailStoreGatewayRedundancyActiveActive/" rel="alternate" type="text/html" title="MailStore Gateway Redundancy - Active-Active configuration" /><published>2023-11-24T00:00:00-07:00</published><updated>2023-11-24T00:00:00-07:00</updated><id>/mailstoregateway/MailStoreGatewayRedundancyActiveActive</id><content type="html" xml:base="/mailstoregateway/MailStoreGatewayRedundancyActiveActive/"><![CDATA[<h2 id="overview-of-the-options-and-risks">Overview of the options and risks</h2>

<p>MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities.</p>

<p>These are unsupported, please do not contact MailStore Support with problems related to these configurations.</p>

<p>Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time.</p>

<p>These are the currently available options:</p>

<h3 id="failover"><a href="/mailstoregateway/MailStoreGatewayRedundancy-Failover.md">Failover</a></h3>

<ul>
  <li>Quick, easy to understand.</li>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Involves short downtimes to switch between servers.</li>
</ul>

<h3 id="failover-with-recovery"><a href="/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery.md">Failover, with recovery</a></h3>

<ul>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Does not require any downtime to switch between servers.</li>
  <li>More technical, requires an understanding of DNS.</li>
</ul>

<h3 id="active-active-this-article"><a href="/mailstoregateway/MailStoreGatewayRedundancy-ActiveActive.md">Active-Active</a> (this article)</h3>

<ul>
  <li>The most complex configuration, and should only be used by experienced administrators.</li>
  <li>Highest risk of message loss due to configuration errors.</li>
  <li>No downtime involved as both servers are running at all times.</li>
  <li>Requires additional configuration within each MailStore instance.</li>
</ul>

<p>This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform)</p>

<h2 id="configuration">Configuration</h2>

<ul>
  <li>
    <p>DNS</p>

    <p>Assuming you have two MailStore Gateway servers, <code class="language-plaintext highlighter-rouge">192.0.2.1</code> and <code class="language-plaintext highlighter-rouge">192.0.2.55</code>, create these DNS records:</p>

    <table>
      <thead>
        <tr>
          <th>Name</th>
          <th>Type</th>
          <th>Record Data</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.1</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.55</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example.</code></td>
          <td><em>MX</em></td>
          <td>10 <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example.</code></td>
          <td><em>MX</em></td>
          <td>20 <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
        </tr>
      </tbody>
    </table>

    <p>To be clear, this configuration has two <em>A records</em>, one pointing to each server, and then an <em>MX record</em> pointing to both servers.</p>

    <p>In this configuration Exchange journal reports will preferentially sent to <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> over <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code>, but you could change the <em>MX records</em> to have the same priority to (roughly) deliver messages equally.</p>

    <p><em>MX records</em> and their ordering are not absolute. Even if you prioritize the servers as described above, you will occasionally see messages delivered to <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code>. This is normal and expected behaviour. Similarly if you configure the <em>MX records</em> to have the same priority the servers will not receive an even 50/50 split of messages. <em>MX records</em> are <em>best effort</em>.</p>
  </li>
  <li>
    <p>Configure <em>MailStore Gateway</em> using a single hostname for all clients</p>

    <p>To be clear, both servers will be configured to have the same name in the MailStore Gateway configuration, e.g. <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code>.</p>

    <p>This is <em>not the same as the DNS records above</em>. In DNS the servers are called <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> and <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code>, but in MailStore Gateway <em>both</em> are configured to respond to <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code>.</p>
  </li>
  <li>
    <p>Backup the configuration</p>

    <p>Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the <a href="https://help.mailstore.com/en/gateway/Backup_and_Restore">Backup and Restore</a> article from the documentation.</p>
  </li>
  <li>
    <p>Keep the configuration backup up to date</p>

    <p>Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes.</p>

    <p><em>Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages. If you change a mailbox password on one server and fail to update the configuration, you will likely lose all messages that are delivered to the second server.</em></p>
  </li>
  <li>
    <p>Go to the first server, configure Let’s Encrypt to <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code></p>
  </li>
  <li>
    <p>Go to the second server, configure Let’s Encrypt to <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code></p>
  </li>
  <li>
    <p>When configuring a MailStore instance to archive messages, create two archiving profiles, one for each server.</p>

    <p>The first profile should be configured to archive server name <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code>, and the second profile should be configured to archive server name <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code>.</p>

    <p>Both will use the same credentials. Both will use <code class="language-plaintext highlighter-rouge">mbx-123@</code>msgw.mailarchiveco.example`, and use the same password.</p>
  </li>
</ul>

<h2 id="failover-process">Failover process</h2>

<p>There isn’t one. Microsoft 365 / Exchange will use the <em>MX records</em> to deliver messages to both servers, and MailStore Gateway will accept messages from both servers. Both servers will receive messages with a <code class="language-plaintext highlighter-rouge">RCPT TO</code> the same address. All MailStore instances pull mail from both gateways at all times.</p>

<h2 id="failover-testing">Failover testing</h2>

<p>Stop the service on either server, and messages will be delivered to the other server and then archived by MailStore normally. The only thing that you’ll notice is that archiving profiles accessing the stopped server will fail and any messages on the stopped server will be held in the queue until you resume operations.</p>

<h2 id="notes">Notes</h2>

<ul>
  <li>
    <p>This process is more complex and less forgiving of errors. You must create a mailbox, replicate the configuration, and only then enable journal rules.</p>
  </li>
  <li>
    <p>You must take care to create mailboxes on one server, then replicate the configuration to the other server.</p>
  </li>
  <li>
    <p>It is not possible to use the web interface to create the same mailbox on more than one server as you cannot assign the mailbox name, nor the password, nor the generated encryption name, you must copy the configuration files manually to reflect changes.</p>
  </li>
  <li>
    <p>Because both servers use identical usernames and passwords, it is possible to move messages between the identically named mailboxes on the respective servers. You should not attempt to do this regularly, but in the case of a server failing catastrophically you could recover the messages from disk.</p>
  </li>
  <li>
    <p>The Gateway servers can be located in different locations, allowing for network or geographic redundancy.</p>
  </li>
</ul>]]></content><author><name></name></author><category term="MailStoreGateway" /><category term="failover " /><summary type="html"><![CDATA[Overview of the options and risks MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities. These are unsupported, please do not contact MailStore Support with problems related to these configurations. Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time. These are the currently available options: Failover Quick, easy to understand. Requires manual intervention to switch between servers. Involves short downtimes to switch between servers. Failover, with recovery Requires manual intervention to switch between servers. Does not require any downtime to switch between servers. More technical, requires an understanding of DNS. Active-Active (this article) The most complex configuration, and should only be used by experienced administrators. Highest risk of message loss due to configuration errors. No downtime involved as both servers are running at all times. Requires additional configuration within each MailStore instance. This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform) Configuration DNS Assuming you have two MailStore Gateway servers, 192.0.2.1 and 192.0.2.55, create these DNS records: Name Type Record Data msgw1.mailarchiveco.example. A 192.0.2.1 msgw2.mailarchiveco.example. A 192.0.2.55 msgw.mailarchiveco.example. MX 10 msgw1.mailarchiveco.example. msgw.mailarchiveco.example. MX 20 msgw2.mailarchiveco.example. To be clear, this configuration has two A records, one pointing to each server, and then an MX record pointing to both servers. In this configuration Exchange journal reports will preferentially sent to msgw1.mailarchiveco.example over msgw2.mailarchiveco.example, but you could change the MX records to have the same priority to (roughly) deliver messages equally. MX records and their ordering are not absolute. Even if you prioritize the servers as described above, you will occasionally see messages delivered to msgw2.mailarchiveco.example. This is normal and expected behaviour. Similarly if you configure the MX records to have the same priority the servers will not receive an even 50/50 split of messages. MX records are best effort. Configure MailStore Gateway using a single hostname for all clients To be clear, both servers will be configured to have the same name in the MailStore Gateway configuration, e.g. msgw.mailarchiveco.example. This is not the same as the DNS records above. In DNS the servers are called msgw1.mailarchiveco.example and msgw2.mailarchiveco.example, but in MailStore Gateway both are configured to respond to msgw.mailarchiveco.example. Backup the configuration Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the Backup and Restore article from the documentation. Keep the configuration backup up to date Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes. Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages. If you change a mailbox password on one server and fail to update the configuration, you will likely lose all messages that are delivered to the second server. Go to the first server, configure Let’s Encrypt to msgw1.mailarchiveco.example Go to the second server, configure Let’s Encrypt to msgw2.mailarchiveco.example When configuring a MailStore instance to archive messages, create two archiving profiles, one for each server. The first profile should be configured to archive server name msgw1.mailarchiveco.example, and the second profile should be configured to archive server name msgw2.mailarchiveco.example. Both will use the same credentials. Both will use mbx-123@msgw.mailarchiveco.example`, and use the same password. Failover process There isn’t one. Microsoft 365 / Exchange will use the MX records to deliver messages to both servers, and MailStore Gateway will accept messages from both servers. Both servers will receive messages with a RCPT TO the same address. All MailStore instances pull mail from both gateways at all times. Failover testing Stop the service on either server, and messages will be delivered to the other server and then archived by MailStore normally. The only thing that you’ll notice is that archiving profiles accessing the stopped server will fail and any messages on the stopped server will be held in the queue until you resume operations. Notes This process is more complex and less forgiving of errors. You must create a mailbox, replicate the configuration, and only then enable journal rules. You must take care to create mailboxes on one server, then replicate the configuration to the other server. It is not possible to use the web interface to create the same mailbox on more than one server as you cannot assign the mailbox name, nor the password, nor the generated encryption name, you must copy the configuration files manually to reflect changes. Because both servers use identical usernames and passwords, it is possible to move messages between the identically named mailboxes on the respective servers. You should not attempt to do this regularly, but in the case of a server failing catastrophically you could recover the messages from disk. The Gateway servers can be located in different locations, allowing for network or geographic redundancy.]]></summary></entry><entry><title type="html">MailStore Gateway Redundancy - Failover with graceful recovery</title><link href="/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery/" rel="alternate" type="text/html" title="MailStore Gateway Redundancy - Failover with graceful recovery" /><published>2023-11-22T00:00:00-07:00</published><updated>2023-11-22T00:00:00-07:00</updated><id>/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery</id><content type="html" xml:base="/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery/"><![CDATA[<h2 id="overview-of-the-options-and-risks">Overview of the options and risks</h2>

<p>MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities.</p>

<p>These are unsupported, please do not contact MailStore Support with problems related to these configurations.</p>

<p>Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time.</p>

<p>These are the currently available options:</p>

<h3 id="failover"><a href="/mailstoregateway/MailStoreGatewayRedundancy-Failover.md">Failover</a></h3>

<ul>
  <li>Quick, easy to understand.</li>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Involves short downtimes to switch between servers.</li>
</ul>

<h3 id="failover-with-recovery-this-article"><a href="/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery.md">Failover, with recovery</a> (this article)</h3>

<ul>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Does not require any downtime to switch between servers.</li>
  <li>More technical, requires an understanding of DNS.</li>
</ul>

<h3 id="active-active"><a href="/mailstoregateway/MailStoreGatewayRedundancy-ActiveActive.md">Active-Active</a></h3>

<ul>
  <li>The most complex configuration, and should only be used by experienced administrators.</li>
  <li>Highest risk of message loss due to configuration errors.</li>
  <li>No downtime involved as both servers are running at all times.</li>
  <li>Requires additional configuration within each MailStore instance.</li>
</ul>

<p>This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform)</p>

<h2 id="configuration">Configuration</h2>

<ul>
  <li>
    <p>DNS</p>

    <p>Assuming  you have two MailStore Gateway servers, <code class="language-plaintext highlighter-rouge">192.0.2.1</code> and <code class="language-plaintext highlighter-rouge">192.0.2.55</code>, create these DNS records:</p>

    <table>
      <thead>
        <tr>
          <th>Name</th>
          <th>Type</th>
          <th>Record Data</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.1</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.55</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example.</code></td>
          <td><em>CNAME</em></td>
          <td><code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
        </tr>
      </tbody>
    </table>

    <p>To be clear, this configuration has two <em>A records</em>, one pointing to each server. It has a <em>CNAME record</em> pointing to the primary server.</p>

    <p>All configuration in MailStore, and on mail servers must only use <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code>, do not reference <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> or <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code> anywhere except these DNS records.</p>
  </li>
  <li>
    <p>Configure <em>MailStore Gateway</em> using a single hostname for all clients</p>

    <p>If your company’s domain is <em>mailarchiveco.example</em> then your gateway’s domain could be <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code>.</p>

    <p>If you have multiple MailStore instances then each would have a separate mailbox, e.g. .<code class="language-plaintext highlighter-rouge">mbx-123...@msgw.mailarchiveco.example</code> and <code class="language-plaintext highlighter-rouge">mbx-456...@msg.mailarchiveco.example</code></p>
  </li>
  <li>
    <p>Backup the configuration</p>

    <p>Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the <a href="https://help.mailstore.com/en/gateway/Backup_and_Restore">Backup and Restore</a> article from the documentation.</p>
  </li>
  <li>
    <p>Keep the configuration backup up to date</p>

    <p>Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes.</p>

    <p><strong>Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages.</strong> If either server has messages in the mailbox when you do a password change you’ll likely lose messages. I would recommend not ever changing the password of the gateway mailbox.</p>
  </li>
  <li>
    <p>You can optionally even have the gateway installed and ready on the other server, just be sure to upgrade both servers together.</p>
  </li>
  <li>
    <p>I would recommend not keeping the service running on the second server, failovers should be done intentionally.</p>

    <p>Technically you can failover automatically (for example, with a monitoring system updating a DNS record or NAT rule) but a human <em>must</em> review the mailbox status on both servers to ensure that no messages were left behind.</p>

    <p>Let’s Encrypt will only work on one server at a time and will fail on the other server.</p>
  </li>
</ul>

<h2 id="failover-process">Failover process</h2>

<p>In the event you have a failure that you cannot resolve within 24-hours:</p>

<ul>
  <li>
    <p>Switch the <em>CNAME record</em> for <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code> from <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> to <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code>.</p>

    <p>This will cause both inbound messages and the archiving profile to use the secondary server.</p>
  </li>
  <li>
    <p>Start the MailStore Gateway on the secondary server.</p>
  </li>
  <li>
    <p>Verify the certificate is valid and properly configured.</p>

    <p>If you use <a href="https://help.mailstore.com/en/gateway/Using_Let%27s_Encrypt_Certificates">Let’s Encrypt</a> then the certificate may be out of date as the secondary server will not be able to renew the certificate. You can manually renew the certificate on the secondary server, but the gateway should update the certificate on start-up.</p>
  </li>
</ul>

<p>In failover mode your DNS records will look like this:</p>

<table>
  <thead>
    <tr>
      <th>Name</th>
      <th>Type</th>
      <th>Record Data</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
      <td><em>A</em></td>
      <td><code class="language-plaintext highlighter-rouge">192.0.2.1</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
      <td><em>A</em></td>
      <td><code class="language-plaintext highlighter-rouge">192.0.2.55</code></td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example.</code></td>
      <td><em>CNAME</em></td>
      <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
    </tr>
  </tbody>
</table>

<h2 id="switching-back-to-the-primary-server">Switching back to the primary server</h2>

<h3 id="recovery-mode">Recovery mode</h3>

<p>Once the primary server is back up and running and you are ready to switch back, there is a new intermediary step involved which is called “recovery mode”.</p>

<ul>
  <li>
    <p>Create a new <em>MX record</em> for <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example</code> pointing to <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code>.</p>

    <p>Your DNS records will now look like this:</p>

    <table>
      <thead>
        <tr>
          <th>Name</th>
          <th>Type</th>
          <th>Record Data</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.1</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
          <td><em>A</em></td>
          <td><code class="language-plaintext highlighter-rouge">192.0.2.55</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
          <td><em>MX</em></td>
          <td><code class="language-plaintext highlighter-rouge">0 msgw1.mailarchiveco.example.</code></td>
        </tr>
        <tr>
          <td><code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example.</code></td>
          <td><em>CNAME</em></td>
          <td><code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></td>
        </tr>
      </tbody>
    </table>

    <p>In this state, inbound messages will be delivered to <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> due to the <em>MX record</em>, while MailStore instances will use the <em>CNAME record</em> to connect to <code class="language-plaintext highlighter-rouge">msgw2.mailarchiveco.example.</code></p>

    <p>Remember to consider DNS TTLs, if you have a 24-hour TTL then you will have to wait 24-hours before the change takes effect globally.</p>
  </li>
  <li>
    <p>Wait for mailboxes on the secondary server to be empty.</p>

    <p>You can check the status of the mailboxes on the secondary server by logging into the MailStore Gateway web interface and checking the status of each mailbox. You can also check the mailbox directories on the secondary server to see if they are empty.</p>
  </li>
  <li>
    <p>Stop the service on the secondary MailStore Gateway.</p>

    <p>It is important to stop the service to prevent the secondary MailStore Gateway from receiving messages, otherwise you may have to repeat the process.</p>
  </li>
  <li>
    <p>Revert your DNS to the original configuration.</p>

    <p>Change the <em>CNAME record</em> back to <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code>, and remove the <em>MX record</em> completely.</p>
  </li>
</ul>

<h2 id="failover-testing">Failover testing</h2>

<p>You can test a failover any time, or switch permanently to the secondary server if desired, just be sure to not leave any messages behind on the non-active server.</p>

<h2 id="notes">Notes</h2>

<p>This process is more complex and potentially less intuitive, and you must have a solid understanding of what is happening. The goal is that in “recovery mode” both server are running, inbound messages are being routed to the primary server, but MailStore is still pulling messages from the secondary server.</p>

<p>Ideally all MailStore instances will check for messages from the gateway frequently, so it should only take a few minutes for the secondary server to be empty. During this time inbound messages are being queued on <code class="language-plaintext highlighter-rouge">msgw1.mailarchiveco.example</code> and will be archived once you return to normal mode.</p>

<p>Remember that messages can sit in the gateway queued for delivery without fear of being lost, so you can take your time with this process.</p>]]></content><author><name></name></author><category term="MailStoreGateway" /><category term="failover " /><summary type="html"><![CDATA[Overview of the options and risks MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities. These are unsupported, please do not contact MailStore Support with problems related to these configurations. Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time. These are the currently available options: Failover Quick, easy to understand. Requires manual intervention to switch between servers. Involves short downtimes to switch between servers. Failover, with recovery (this article) Requires manual intervention to switch between servers. Does not require any downtime to switch between servers. More technical, requires an understanding of DNS. Active-Active The most complex configuration, and should only be used by experienced administrators. Highest risk of message loss due to configuration errors. No downtime involved as both servers are running at all times. Requires additional configuration within each MailStore instance. This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform) Configuration DNS Assuming you have two MailStore Gateway servers, 192.0.2.1 and 192.0.2.55, create these DNS records: Name Type Record Data msgw1.mailarchiveco.example. A 192.0.2.1 msgw2.mailarchiveco.example. A 192.0.2.55 msgw.mailarchiveco.example. CNAME msgw1.mailarchiveco.example. To be clear, this configuration has two A records, one pointing to each server. It has a CNAME record pointing to the primary server. All configuration in MailStore, and on mail servers must only use msgw.mailarchiveco.example, do not reference msgw1.mailarchiveco.example or msgw2.mailarchiveco.example anywhere except these DNS records. Configure MailStore Gateway using a single hostname for all clients If your company’s domain is mailarchiveco.example then your gateway’s domain could be msgw.mailarchiveco.example. If you have multiple MailStore instances then each would have a separate mailbox, e.g. .mbx-123...@msgw.mailarchiveco.example and mbx-456...@msg.mailarchiveco.example Backup the configuration Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the Backup and Restore article from the documentation. Keep the configuration backup up to date Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes. Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages. If either server has messages in the mailbox when you do a password change you’ll likely lose messages. I would recommend not ever changing the password of the gateway mailbox. You can optionally even have the gateway installed and ready on the other server, just be sure to upgrade both servers together. I would recommend not keeping the service running on the second server, failovers should be done intentionally. Technically you can failover automatically (for example, with a monitoring system updating a DNS record or NAT rule) but a human must review the mailbox status on both servers to ensure that no messages were left behind. Let’s Encrypt will only work on one server at a time and will fail on the other server. Failover process In the event you have a failure that you cannot resolve within 24-hours: Switch the CNAME record for msgw.mailarchiveco.example from msgw1.mailarchiveco.example to msgw2.mailarchiveco.example. This will cause both inbound messages and the archiving profile to use the secondary server. Start the MailStore Gateway on the secondary server. Verify the certificate is valid and properly configured. If you use Let’s Encrypt then the certificate may be out of date as the secondary server will not be able to renew the certificate. You can manually renew the certificate on the secondary server, but the gateway should update the certificate on start-up. In failover mode your DNS records will look like this: Name Type Record Data msgw1.mailarchiveco.example. A 192.0.2.1 msgw2.mailarchiveco.example. A 192.0.2.55 msgw.mailarchiveco.example. CNAME msgw2.mailarchiveco.example. Switching back to the primary server Recovery mode Once the primary server is back up and running and you are ready to switch back, there is a new intermediary step involved which is called “recovery mode”. Create a new MX record for msgw2.mailarchiveco.example pointing to msgw1.mailarchiveco.example. Your DNS records will now look like this: Name Type Record Data msgw1.mailarchiveco.example. A 192.0.2.1 msgw2.mailarchiveco.example. A 192.0.2.55 msgw2.mailarchiveco.example. MX 0 msgw1.mailarchiveco.example. msgw.mailarchiveco.example. CNAME msgw2.mailarchiveco.example. In this state, inbound messages will be delivered to msgw1.mailarchiveco.example due to the MX record, while MailStore instances will use the CNAME record to connect to msgw2.mailarchiveco.example. Remember to consider DNS TTLs, if you have a 24-hour TTL then you will have to wait 24-hours before the change takes effect globally. Wait for mailboxes on the secondary server to be empty. You can check the status of the mailboxes on the secondary server by logging into the MailStore Gateway web interface and checking the status of each mailbox. You can also check the mailbox directories on the secondary server to see if they are empty. Stop the service on the secondary MailStore Gateway. It is important to stop the service to prevent the secondary MailStore Gateway from receiving messages, otherwise you may have to repeat the process. Revert your DNS to the original configuration. Change the CNAME record back to msgw1.mailarchiveco.example, and remove the MX record completely. Failover testing You can test a failover any time, or switch permanently to the secondary server if desired, just be sure to not leave any messages behind on the non-active server. Notes This process is more complex and potentially less intuitive, and you must have a solid understanding of what is happening. The goal is that in “recovery mode” both server are running, inbound messages are being routed to the primary server, but MailStore is still pulling messages from the secondary server. Ideally all MailStore instances will check for messages from the gateway frequently, so it should only take a few minutes for the secondary server to be empty. During this time inbound messages are being queued on msgw1.mailarchiveco.example and will be archived once you return to normal mode. Remember that messages can sit in the gateway queued for delivery without fear of being lost, so you can take your time with this process.]]></summary></entry><entry><title type="html">MailStore Gateway Redundancy - Failover</title><link href="/mailstoregateway/MailStoreGatewayRedundancy-Failover/" rel="alternate" type="text/html" title="MailStore Gateway Redundancy - Failover" /><published>2023-11-20T00:00:00-07:00</published><updated>2023-11-20T00:00:00-07:00</updated><id>/mailstoregateway/MailStoreGatewayRedundancy-Failover</id><content type="html" xml:base="/mailstoregateway/MailStoreGatewayRedundancy-Failover/"><![CDATA[<h2 id="overview-of-the-options-and-risks">Overview of the options and risks</h2>

<p>MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities.</p>

<p>These are unsupported, please do not contact MailStore Support with problems related to these configurations.</p>

<p>Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time.</p>

<p>These are the currently available options:</p>

<h3 id="failover-this-article"><a href="/mailstoregateway/MailStoreGatewayRedundancy-Failover.md">Failover</a> (this article)</h3>

<ul>
  <li>Quick, easy to understand.</li>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Involves short downtimes to switch between servers.</li>
</ul>

<h3 id="failover-with-recovery"><a href="/mailstoregateway/MailStoreGatewayRedundancy-FailoverWithRecovery.md">Failover, with recovery</a></h3>

<ul>
  <li>Requires manual intervention to switch between servers.</li>
  <li>Does not require any downtime to switch between servers.</li>
  <li>More technical, requires an understanding of DNS.</li>
</ul>

<h3 id="active-active"><a href="/mailstoregateway/MailStoreGatewayRedundancy-ActiveActive.md">Active-Active</a></h3>

<ul>
  <li>The most complex configuration, and should only be used by experienced administrators.</li>
  <li>Highest risk of message loss due to configuration errors.</li>
  <li>No downtime involved as both servers are running at all times.</li>
  <li>Requires additional configuration within each MailStore instance.</li>
</ul>

<p>This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform)</p>

<h2 id="configuration">Configuration</h2>

<ul>
  <li>
    <p>Configure <em>MailStore Gateway</em> using a single hostname for all clients</p>

    <p>If your company’s domain is <em>mailarchiveco.example</em> then your gateway could be <code class="language-plaintext highlighter-rouge">msgw.mailarchiveco.example</code>, if you have multiple MailStore instances then each would have a separate mailbox, e.g. <code class="language-plaintext highlighter-rouge">mbx-123...@mailarchiveco.example</code> and <code class="language-plaintext highlighter-rouge">mbx-456...@mailarchiveco.example</code>.</p>
  </li>
  <li>
    <p>Backup the configuration</p>

    <p>Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the <a href="https://help.mailstore.com/en/gateway/Backup_and_Restore">Backup and Restore</a> article from the documentation.</p>
  </li>
  <li>
    <p>Keep the configuration backup up to date</p>

    <p>Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes.</p>

    <p><strong>Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages.</strong> If either server has messages in the mailbox when you do a password change you’ll likely lose messages. I would recommend not ever changing the password of the gateway mailbox.</p>
  </li>
  <li>
    <p>You can optionally even have the gateway installed and ready, just be sure to upgrade both servers to matching versions of MailStore Gateway.</p>
  </li>
  <li>
    <p>I would recommend not keeping the service running on the second server, failovers should be done intentionally.</p>

    <p>Technically you can failover automatically (for example, with a monitoring system updating a DNS record or NAT rule) but a human <em>must</em> review the mailbox status on both servers to ensure that no messages were left behind.</p>

    <p>Let’s Encrypt will only work on one server at a time and will fail on the other server.</p>
  </li>
</ul>

<h2 id="failover-process">Failover process</h2>

<p>In the event you have a failure that you cannot resolve within 24-hours:</p>

<ul>
  <li>
    <p>Either switch the DNS <em>A record</em> or update your NAT firewall rules to redirect traffic to the secondary server.</p>
  </li>
  <li>
    <p>Start MailStore Gateway on the secondary server.</p>
  </li>
  <li>
    <p>Verify the certificate is valid and properly configured.</p>

    <p>If you use <a href="https://help.mailstore.com/en/gateway/Using_Let%27s_Encrypt_Certificates">Let’s Encrypt</a> then the certificate may be out of date as the secondary server will not be able to renew the certificate. You can manually renew the certificate on the secondary server, but the gateway should update the certificate on start-up.</p>
  </li>
</ul>

<h2 id="switching-back-to-the-primary-server">Switching back to the primary server</h2>

<p>Once the primary server is back up and running, you can switch back to it by reversing the DNS <em>A record</em> or NAT firewall rules, then review the mailbox status on both servers.</p>

<ul>
  <li>
    <p>Switch the DNS <em>A record</em> or NAT firewall rule back to the primary server.</p>

    <p>Remember to consider DNS TTLs, if you have a 24-hour TTL then you will have to wait 24-hours before the change takes effect globally.</p>
  </li>
  <li>
    <p>Stop the service on the secondary MailStore Gateway.</p>

    <p>It is important to stop the service to prevent the secondary MailStore Gateway from receiving messages, otherwise you may have to repeat the process.</p>
  </li>
  <li>
    <p>Move the mailbox directories containing the messages from the secondary server to the primary server. The mailbox directories will already exist, you are just adding the files from one into the other.</p>
  </li>
</ul>

<h2 id="failover-testing">Failover testing</h2>

<p>You can test a failover any time, or switch permanently to the secondary server if desired, just be sure to not leave any messages behind on the non-active server.</p>]]></content><author><name></name></author><category term="MailStoreGateway" /><category term="failover " /><summary type="html"><![CDATA[Overview of the options and risks MailStore Gateway does not offer any high availability capabilities, but there are a few options to provide redundancy depending on your requirements and technical abilities. These are unsupported, please do not contact MailStore Support with problems related to these configurations. Consider if you need redundancy at all. A short term outage of a gateway is not an emergency and will not cause message loss. Journal reports are e-mails like any other and the sending server will queue them for re-delivery for period of time. These are the currently available options: Failover (this article) Quick, easy to understand. Requires manual intervention to switch between servers. Involves short downtimes to switch between servers. Failover, with recovery Requires manual intervention to switch between servers. Does not require any downtime to switch between servers. More technical, requires an understanding of DNS. Active-Active The most complex configuration, and should only be used by experienced administrators. Highest risk of message loss due to configuration errors. No downtime involved as both servers are running at all times. Requires additional configuration within each MailStore instance. This article assumes a basic understanding of how to use MailStore Gateway, networking, DNS, and Microsoft 365 (or your e-mail platform) Configuration Configure MailStore Gateway using a single hostname for all clients If your company’s domain is mailarchiveco.example then your gateway could be msgw.mailarchiveco.example, if you have multiple MailStore instances then each would have a separate mailbox, e.g. mbx-123...@mailarchiveco.example and mbx-456...@mailarchiveco.example. Backup the configuration Once the primary server is configured, take a backup of your MailStore Gateway configuration as described in the Backup and Restore article from the documentation. Keep the configuration backup up to date Be sure to update the configuration backup whenever you make any configuration changes to the gateway. This applies to new mailboxes as well as password changes. Warning: MailStore Gateway stores all messages encrypted at rest, relying on the mailbox password to decrypt messages. If either server has messages in the mailbox when you do a password change you’ll likely lose messages. I would recommend not ever changing the password of the gateway mailbox. You can optionally even have the gateway installed and ready, just be sure to upgrade both servers to matching versions of MailStore Gateway. I would recommend not keeping the service running on the second server, failovers should be done intentionally. Technically you can failover automatically (for example, with a monitoring system updating a DNS record or NAT rule) but a human must review the mailbox status on both servers to ensure that no messages were left behind. Let’s Encrypt will only work on one server at a time and will fail on the other server. Failover process In the event you have a failure that you cannot resolve within 24-hours: Either switch the DNS A record or update your NAT firewall rules to redirect traffic to the secondary server. Start MailStore Gateway on the secondary server. Verify the certificate is valid and properly configured. If you use Let’s Encrypt then the certificate may be out of date as the secondary server will not be able to renew the certificate. You can manually renew the certificate on the secondary server, but the gateway should update the certificate on start-up. Switching back to the primary server Once the primary server is back up and running, you can switch back to it by reversing the DNS A record or NAT firewall rules, then review the mailbox status on both servers. Switch the DNS A record or NAT firewall rule back to the primary server. Remember to consider DNS TTLs, if you have a 24-hour TTL then you will have to wait 24-hours before the change takes effect globally. Stop the service on the secondary MailStore Gateway. It is important to stop the service to prevent the secondary MailStore Gateway from receiving messages, otherwise you may have to repeat the process. Move the mailbox directories containing the messages from the secondary server to the primary server. The mailbox directories will already exist, you are just adding the files from one into the other. Failover testing You can test a failover any time, or switch permanently to the secondary server if desired, just be sure to not leave any messages behind on the non-active server.]]></summary></entry><entry><title type="html">Decrypt and access messages in the MailStore Gateway mailbox via curl</title><link href="/mailstoregateway/MailStoreGatewayAccessViaCurl/" rel="alternate" type="text/html" title="Decrypt and access messages in the MailStore Gateway mailbox via curl" /><published>2023-10-01T00:00:00-06:00</published><updated>2023-10-01T00:00:00-06:00</updated><id>/mailstoregateway/MailStoreGatewayAccessViaCurl</id><content type="html" xml:base="/mailstoregateway/MailStoreGatewayAccessViaCurl/"><![CDATA[<p>MailStore Gateway includes a POP3 server as part of its proxy functionality. This feature allows access to messages in the mailbox, enabling you to inspect messages directly using a tool like <em>curl</em>. This is useful as it allows you to access decrypted messages.</p>

<p>You will need the mailbox’s name and password. As the password is part of the message decryption process, it is not possible to access the mailbox or messages contained within without the password. Additionally, be aware that a password reset is not possible while the mailbox contains messages.</p>

<p>In order to access a message, you first need to get the message number from the message list, so we will start by connecting to the POP3 server and requesting the message list.</p>

<p>Use the following <em>curl</em> command:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --ssl-reqd --insecure pop3://msgw.example.com/ -u "mbx-2a20342e96294505a35eb5c8364eb67c"
</code></pre></div></div>

<p>The response will look like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1 91471
</code></pre></div></div>

<p>Now try doing a UIDL request as well:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --ssl-reqd --insecure --request UIDL pop3://msgw.example.com/ -u "mbx-2a20342e96294505a35eb5c8364eb67c"
</code></pre></div></div>

<p>The response will look like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1 f558587b-b6d6-490d-b21e-eaf02172131a
</code></pre></div></div>

<p>In both cases above, there is a single message in the mailbox, the size is <em>91471 bytes</em> and UIDL is <em>f558587b-b6d6-490d-b21e-eaf02172131a</em>.</p>

<p>In the POP3 protocol messages are always numbered starting at 1, and message numbers may change between sessions. While the POP3 protocol does not specify how to generate UIDLs it does require that the UIDL is consistent. MailStore Gateway uses the filename on disk as the UIDL making it possible to identify the message in the filesystem.</p>

<p>Once you have the message number, you can download it by adding the message number to the URI:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl --ssl-reqd --insecure pop3://msgw.example.com/1 -u "mbx-2a20342e96294505a35eb5c8364eb67c"
</code></pre></div></div>

<p>Technically the message numbers can change from session to session, so if there are new messages coming in and/or messages being deleted you might need to try a few times, and you should verify using the Date header or Received headers of the message to make sure it matches.</p>

<p>Normally in POP3 you would pull the list of messages and then retrieve the message in the same session (numbers are consistent within a session but are allowed to change between sessions) but curl doesn’t support this, so you will need to do it in two steps.</p>

<p>Modern versions of Windows have curl built in, as well as nearly any Linux environment (including WSL) so it is quick and easy to get going, and by default it leaves messages on the server whereas most other POP3 clients default to deleting messages after they’re accessed.</p>

<p>MailStore Gateway’s passwords are generated randomly and include punctuation, so it is easiest to paste it into curl each time. As an example, a password may include punctuation such as <code class="language-plaintext highlighter-rouge">qn5x(/.B9M~PsZA92f]VYbIG\q{!1^[8</code> which would require an understanding of shell escape sequences to enter safely.</p>]]></content><author><name></name></author><category term="MailStoreGateway" /><category term="data management" /><summary type="html"><![CDATA[MailStore Gateway includes a POP3 server as part of its proxy functionality. This feature allows access to messages in the mailbox, enabling you to inspect messages directly using a tool like curl. This is useful as it allows you to access decrypted messages. You will need the mailbox’s name and password. As the password is part of the message decryption process, it is not possible to access the mailbox or messages contained within without the password. Additionally, be aware that a password reset is not possible while the mailbox contains messages. In order to access a message, you first need to get the message number from the message list, so we will start by connecting to the POP3 server and requesting the message list. Use the following curl command: curl --ssl-reqd --insecure pop3://msgw.example.com/ -u "mbx-2a20342e96294505a35eb5c8364eb67c" The response will look like this: 1 91471 Now try doing a UIDL request as well: curl --ssl-reqd --insecure --request UIDL pop3://msgw.example.com/ -u "mbx-2a20342e96294505a35eb5c8364eb67c" The response will look like this: 1 f558587b-b6d6-490d-b21e-eaf02172131a In both cases above, there is a single message in the mailbox, the size is 91471 bytes and UIDL is f558587b-b6d6-490d-b21e-eaf02172131a. In the POP3 protocol messages are always numbered starting at 1, and message numbers may change between sessions. While the POP3 protocol does not specify how to generate UIDLs it does require that the UIDL is consistent. MailStore Gateway uses the filename on disk as the UIDL making it possible to identify the message in the filesystem. Once you have the message number, you can download it by adding the message number to the URI: curl --ssl-reqd --insecure pop3://msgw.example.com/1 -u "mbx-2a20342e96294505a35eb5c8364eb67c" Technically the message numbers can change from session to session, so if there are new messages coming in and/or messages being deleted you might need to try a few times, and you should verify using the Date header or Received headers of the message to make sure it matches. Normally in POP3 you would pull the list of messages and then retrieve the message in the same session (numbers are consistent within a session but are allowed to change between sessions) but curl doesn’t support this, so you will need to do it in two steps. Modern versions of Windows have curl built in, as well as nearly any Linux environment (including WSL) so it is quick and easy to get going, and by default it leaves messages on the server whereas most other POP3 clients default to deleting messages after they’re accessed. MailStore Gateway’s passwords are generated randomly and include punctuation, so it is easiest to paste it into curl each time. As an example, a password may include punctuation such as qn5x(/.B9M~PsZA92f]VYbIG\q{!1^[8 which would require an understanding of shell escape sequences to enter safely.]]></summary></entry><entry><title type="html">Explaining MailStore’s folder structure</title><link href="/mailstoreserver/mailstorespe/mailstorehome/Explaining-MailStore's-folder-structure/" rel="alternate" type="text/html" title="Explaining MailStore’s folder structure" /><published>2023-02-14T00:00:00-07:00</published><updated>2023-02-14T00:00:00-07:00</updated><id>/mailstoreserver/mailstorespe/mailstorehome/Explaining%20MailStore&apos;s%20folder%20structure</id><content type="html" xml:base="/mailstoreserver/mailstorespe/mailstorehome/Explaining-MailStore&apos;s-folder-structure/"><![CDATA[<p>All messages in MailStore are within an individual user’s archive, each user has a separate archive, so for example my incoming messages could end up in any/all of these depending on where the messages come from:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Archive of dwarren\Exchange dwarren\Inbox
Archive of dwarren\Journal Incoming
Archive of dwarren\dwarren@example.com\Inbox
</code></pre></div></div>

<p>I will skip typing “Archive of dwarren” going forward as it applies identically in all cases below.</p>

<p>In this context, MailStore’s archiving profiles can be broken down into two types that I’ll discuss individually, <em>Individual User</em> and <em>Multiple User</em>.</p>

<p><em>Individual User</em> profiles are where there is a 1:1 relationship between the source mailbox and the owner of the messages in MailStore does not parse the headers to determine the owner. These start by knowing the target archive (user), then connect to a mailbox to find messages.  This includes Multiple Mailbox profiles such as Exchange Multiple Mailbox, as this type of profile is really just a series of Single Mailbox profiles, with a 1:1 relationship between each source mailbox and target archive.</p>

<p>For Individual User profiles, each message is archived into only a single user (if you send a message, a separate copy will be retrieved from your Sent folder, but a copy is not placed in the recipient’s archive, instead a copy must be found in their mailbox for it to appear in their archive in MailStore For these profiles the folder structure is preserved so you’ll get folder structures that look something like this:</p>

<p>Exchange mailboxes (Single and Multiple mailbox, but not Gateway or Journal):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Exchange $MAILBOXNAME$\Inbox
Exchange $MAILBOXNAME$\Sent Items
Exchange $MAILBOXNAME$\And more!
</code></pre></div></div>

<p>IMAP mailboxes:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$EMAILADDRESS$\INBOX
$EMAILADDRESS$\Sent
$EMAILADDRESS$\And more!
</code></pre></div></div>

<p>Local files and mail prepend the source (Outlook is both Outlook itself and PST files) and then mirror the folder structure.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Outlook $PSTTITLE\Inbox
Outlook $PSTTITLE\Sent Items
Outlook $PSTTITLE\And more!
</code></pre></div></div>

<p>By “And More!” I mean all folders the user has created or otherwise exist in the mailbox.</p>

<p>One quirk, Google doesn’t have folders in the classic sense, and their Labels implementation is not compatible with the way MailStore organizes and deduplicates messages, so instead messages are sorted into only the Inbox and Send folder:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$EMAILADDRESS$\Inbox
$EMAILADDRESS$\Sent
</code></pre></div></div>

<p>“Multiple User” type profiles have a “1:many” relationship, MailStore parses the headers of each message to determine the sender and recipients. These are Gateway/Journal/Multi-drop (mail belonging to multiple individuals is located in a single mailbox and MailStore must therefore parse the headers to determine the owner). MailStore starts with a message, then determines the target archive by parsing the sender and recipient(s), looking for matching e-mail addresses in MailStore’s user list and placing a copy in each. Messages can have 0 or more target archives (for example, if you send a message to two co-workers, the single message can be archived into all three archives from one instance).</p>

<p>For Gateway/Journal/Multi-drop profiles the folder structure is simpler:</p>

<p>Exchange mailboxes target the “Journal” folders:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Journal Incoming
Journal Outgoing
</code></pre></div></div>

<p>IMAP and local files use the e-mail address plus a \ as the root.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$EMAILADDRESS$\INBOX
$EMAILADDRESS$\Sent Items
</code></pre></div></div>

<p>In these cases there is no further folder structure possible, MailStore is sorting messages internally (in most cases all the messages came from the source’s inbox folder).</p>

<p>The one quirk here, you can have messages in the source that have no matching user in MailStore, in this case the message can either be targeted to the “@catchall\Unknown E-mail Addresses” or skipped and left in the source mailbox.</p>

<p>A couple other things to be aware of, within a target scope (The “Archive of” plus one more layer, so “Archive of dwarren\Exchange” and everything below, only one instance of a single message is allowed and subsequent copies will be skipped – So if you copy a message from your Inbox to your “Important Stuff” folder on the server, in MailStore it is just in Inbox. When you delete it from your mailboxes’ Inbox MailStore will now find the copy in your mailbox’s “Important Stuff” folder and move the message in MailStore to match.</p>

<p>Lastly, this all applies to how MailStore presents message in the interface. There is a single-instance-storage system that deduplicates on disk for storage savings as well. The single instance storage system is totally disconnected from where/how messages are stored in the interface, MailStore splits messages at the MIME part level and stores each MIME block once.</p>]]></content><author><name></name></author><category term="MailStoreServer" /><category term="MailStoreSPE" /><category term="MailStoreHome" /><category term="data management" /><summary type="html"><![CDATA[All messages in MailStore are within an individual user’s archive, each user has a separate archive, so for example my incoming messages could end up in any/all of these depending on where the messages come from: Archive of dwarren\Exchange dwarren\Inbox Archive of dwarren\Journal Incoming Archive of dwarren\dwarren@example.com\Inbox I will skip typing “Archive of dwarren” going forward as it applies identically in all cases below. In this context, MailStore’s archiving profiles can be broken down into two types that I’ll discuss individually, Individual User and Multiple User. Individual User profiles are where there is a 1:1 relationship between the source mailbox and the owner of the messages in MailStore does not parse the headers to determine the owner. These start by knowing the target archive (user), then connect to a mailbox to find messages. This includes Multiple Mailbox profiles such as Exchange Multiple Mailbox, as this type of profile is really just a series of Single Mailbox profiles, with a 1:1 relationship between each source mailbox and target archive. For Individual User profiles, each message is archived into only a single user (if you send a message, a separate copy will be retrieved from your Sent folder, but a copy is not placed in the recipient’s archive, instead a copy must be found in their mailbox for it to appear in their archive in MailStore For these profiles the folder structure is preserved so you’ll get folder structures that look something like this: Exchange mailboxes (Single and Multiple mailbox, but not Gateway or Journal): Exchange $MAILBOXNAME$\Inbox Exchange $MAILBOXNAME$\Sent Items Exchange $MAILBOXNAME$\And more! IMAP mailboxes: $EMAILADDRESS$\INBOX $EMAILADDRESS$\Sent $EMAILADDRESS$\And more! Local files and mail prepend the source (Outlook is both Outlook itself and PST files) and then mirror the folder structure. Outlook $PSTTITLE\Inbox Outlook $PSTTITLE\Sent Items Outlook $PSTTITLE\And more! By “And More!” I mean all folders the user has created or otherwise exist in the mailbox. One quirk, Google doesn’t have folders in the classic sense, and their Labels implementation is not compatible with the way MailStore organizes and deduplicates messages, so instead messages are sorted into only the Inbox and Send folder: $EMAILADDRESS$\Inbox $EMAILADDRESS$\Sent “Multiple User” type profiles have a “1:many” relationship, MailStore parses the headers of each message to determine the sender and recipients. These are Gateway/Journal/Multi-drop (mail belonging to multiple individuals is located in a single mailbox and MailStore must therefore parse the headers to determine the owner). MailStore starts with a message, then determines the target archive by parsing the sender and recipient(s), looking for matching e-mail addresses in MailStore’s user list and placing a copy in each. Messages can have 0 or more target archives (for example, if you send a message to two co-workers, the single message can be archived into all three archives from one instance). For Gateway/Journal/Multi-drop profiles the folder structure is simpler: Exchange mailboxes target the “Journal” folders: Journal Incoming Journal Outgoing IMAP and local files use the e-mail address plus a \ as the root. $EMAILADDRESS$\INBOX $EMAILADDRESS$\Sent Items In these cases there is no further folder structure possible, MailStore is sorting messages internally (in most cases all the messages came from the source’s inbox folder). The one quirk here, you can have messages in the source that have no matching user in MailStore, in this case the message can either be targeted to the “@catchall\Unknown E-mail Addresses” or skipped and left in the source mailbox. A couple other things to be aware of, within a target scope (The “Archive of” plus one more layer, so “Archive of dwarren\Exchange” and everything below, only one instance of a single message is allowed and subsequent copies will be skipped – So if you copy a message from your Inbox to your “Important Stuff” folder on the server, in MailStore it is just in Inbox. When you delete it from your mailboxes’ Inbox MailStore will now find the copy in your mailbox’s “Important Stuff” folder and move the message in MailStore to match. Lastly, this all applies to how MailStore presents message in the interface. There is a single-instance-storage system that deduplicates on disk for storage savings as well. The single instance storage system is totally disconnected from where/how messages are stored in the interface, MailStore splits messages at the MIME part level and stores each MIME block once.]]></summary></entry><entry><title type="html">Removing duplicates</title><link href="/mailstoreserver/mailstorespe/mailstorehome/Removing-duplicates/" rel="alternate" type="text/html" title="Removing duplicates" /><published>2023-02-07T00:00:00-07:00</published><updated>2023-02-07T00:00:00-07:00</updated><id>/mailstoreserver/mailstorespe/mailstorehome/Removing%20duplicates</id><content type="html" xml:base="/mailstoreserver/mailstorespe/mailstorehome/Removing-duplicates/"><![CDATA[<h2 id="introduction">Introduction</h2>

<p>MailStore does not have any formal deduplication feature.</p>

<p>While archiving MailStore checks each message in the source mailbox to see if it is in the archive in the correct location, skipping messages already in the archive, so in normal operation you should not see duplicated messages.</p>

<p>However, if a user or a mailbox is renamed, you might end up with two different sets of folders for the same user, and although you can merge these together, if the two folders contain the same messages you’ll end up with duplicates.</p>

<p>Because MailStore’s archiving process skips messages already in the archive we can export, delete and re-archive the messages, skipping the duplicated messages in most cases.</p>

<p>You must first finish merging and organizing messages in MailStore before proceeding.</p>

<h2 id="account">Account</h2>

<p>You can perform these tasks as the <em>admin</em>, but you need to unblock access to the archive under <em>Administrative Tools</em> -&gt; <em>Compliance</em> -&gt; <em>Compliance General</em>.</p>

<p>Alternatively an individual user can perform most of these steps. The user account must have <em>Archive E-mail</em>, <em>Export E-mail</em>, <em>Delete</em> permissions on their account and also <em>Full Access</em> to their archive. The admin will still need to be involved with certain steps, so it is best to perform the following steps as an administrator.</p>

<p>Login to MailStore using whatever account you like, all of the steps will be performed in <em>MailStore Client</em> using the same account other than any <em>Administrative Tools</em>  steps must be performed by an admin.</p>

<h2 id="select-the-messages">Select the messages</h2>

<p>This step is optional, but in some cases you’ll be able to narrow down the range of messages using the search filter. If this is not possible, you can perform the following steps on individual folders, entire user archives, or across the entire MailStore archive as you prefer.</p>

<p>Start by finding a few examples of duplicate messages, they should already be in the same folder and will have the same <em>Date</em>, but they may be archived on different dates, if so the goal is to determine the date that the duplication occurred. You can select a wider date range if you like.</p>

<ol>
  <li>Go to <em>Search E-mail</em></li>
  <li>Click the <em>New Query</em> button.</li>
  <li>Optional: Click the Folder <em>[…]</em> button and select a starting point. You can leave this blank to work across the entire archive.</li>
  <li>Set the <em>Date</em> field to find messages older than the date the duplication occurred.</li>
  <li>Set the <em>Archived Date</em> field to find messages <em>not older than</em> than the date the duplication occurred.</li>
  <li>Click the <em>Create Search Folder</em>, save the search.</li>
</ol>

<p>You can review the search if you want, hopefully it will contain one instance of each duplicated message but it may contain both copies of the duplicated messages and/or it may contain messages that are not duplicated. This is fine. The goal is to make sure that at least one instance of every duplicated message is found.</p>

<h2 id="stop-mailstore-archive-e-mail-profiles-from-running">Stop MailStore Archive E-mail profiles from running</h2>

<p>It is highly recommend to stop all archiving tasks for the next steps. This can be done in more than one way:</p>

<ul>
  <li>Launch the <em>Configure the MailStore Server Service</em> tool.</li>
  <li>Stop the service.</li>
  <li>Start in <em>Safe Mode</em>.</li>
</ul>

<p>This will stop all background tasks, and only allow <em>admin</em> users to login</p>

<p>-OR-</p>

<ul>
  <li>Go to the <em>Archive E-mail</em> panel</li>
  <li>Select <em>Show Profiles of All Users</em> at the bottom.</li>
  <li>Select each of the profiles that have a spinning circle, right click, select <em>Manual</em>.</li>
</ul>

<p>Whatever step you take, be sure to keep track of what you changed so you can revert the change later.</p>

<h2 id="export-the-messages">Export the messages</h2>

<ol>
  <li>Go to <em>Export E-mail</em></li>
  <li>Create a new <em>E-mail Files</em> export, <em>Directory (File System)</em>, type <em>EML files</em></li>
  <li>Set the <em>Scope</em> to the messages to be deduplicated, this can be the saved search folder created in the previous step or any other folder you like. Subfolders are included.</li>
  <li>Set the <em>Target Folder</em> to a local directory that has enough space. I’ll use <code class="language-plaintext highlighter-rouge">C:\EML</code>. The directory will be created as needed.</li>
  <li>Enable both <em>Retain folder structure</em> and <em>Update existing export</em>.</li>
  <li>Complete the profile and <em>Run</em>.</li>
</ol>

<p>A note about the folder you select:</p>

<p>While most operations in MailStore Server use folders relative to the server, in this case we’re using a folder relative to the client. So for example, if MailStore Client is installed on your personal workstation and you export to <code class="language-plaintext highlighter-rouge">C:\EML</code> the files will be written to your workstation instead of the server.</p>

<p>It is possible to use a mapped network drive, UNC path, or external drive, but I would recommend finding a machine with sufficient local internal storage instead as this process is dependent on your disk performance.</p>

<h2 id="delete-the-messages">Delete the messages</h2>

<p>Once the export finishes, return to <em>MailStore Client</em> and delete the messages.</p>

<h3 id="if-you-exported-the-entire-mailstore-archive">If you exported the entire MailStore archive</h3>

<ol>
  <li>Go to <em>Administrative Tools</em> -&gt; <em>Storage</em> -&gt; <em>Storage Locations</em></li>
  <li>Right click on each Storage Location and <em>Detach</em></li>
  <li>Create a new Storage Location. MailStore’s default name is in the format <em>YYYY-MM</em>, I would recommend staying with this format.</li>
  <li>Set it to <em>Archive Here</em>.</li>
</ol>

<h3 id="if-you-used-a-saved-search-folder">If you used a saved search folder</h3>

<ol>
  <li>Navigate to the saved search.</li>
  <li>Select the top-most message (single click).</li>
  <li>Scroll to the bottom, <em>Shift-Click</em> the bottom-most message to highlight the range.</li>
  <li>Right click on any selected message and select <em>Delete</em>.</li>
</ol>

<h3 id="if-you-exported-any-other-folder">If you exported any other folder</h3>

<ol>
  <li>Right click on the folder.</li>
  <li>Select <em>Delete</em>.</li>
</ol>

<h2 id="re-archive-any-missing-messages">Re-archive any missing messages</h2>

<p>Now we’ll re-archive the EML files that we exported.</p>

<ol>
  <li>Go to <em>Archive E-mail</em>.</li>
  <li>Select <em>E-mail Files</em>.</li>
  <li>Select <em>EML and MSG files</em>.</li>
  <li>Select type <em>MailStore Export</em>.</li>
  <li>Set the folder (<code class="language-plaintext highlighter-rouge">C:\EML</code> in my case).</li>
  <li>Leave other options at their defaults (<em>Include Subfolders</em>, <em>Read MailStore Headers</em> enabled, <em>Verify Signature</em> disabled).</li>
  <li>Click <em>Next &gt;</em></li>
  <li>Leave all <em>Advanced Settings</em> at their defaults.</li>
  <li>Click <em>Next &gt;</em></li>
  <li>Under <em>Target Archive</em> select the <em>admin</em> user, or your own user. This archive is not used, but the field must be set.</li>
  <li>Complete the profile and <em>Run</em>.</li>
</ol>

<p>This will re-archive all of the exported messages, skipping messages already in the archive. If you were able to use a <em>Search E-mail</em> folder to select just a subset of messages then most messages will be skipped but there may be a few messages that were in the selected date ranges that did not get duplicated and we want to make sure we get these messages re-archived.</p>

<p>It is possible that you won’t have any messages get archived here at all.</p>

<p>When the profile finishes click the <em>Details</em> link, all messages should either be archived, or skipped because they were already in the archive.</p>

<h2 id="cleanup">Cleanup</h2>

<h3 id="re-enable-archiving">Re-enable archiving</h3>

<p>You can now return MailStore to normal operation by undoing whatever steps you took in the <em>Stop MailStore Archive E-mail profiles from running</em> step.</p>

<h3 id="free-up-disk-space">Free up disk space</h3>

<p>Depending on how you deleted the messages, you might be able to get some disk space back.</p>

<h4 id="deleted-all-storage-locations">Deleted all Storage Locations</h4>

<p>If you deleted all of your <em>Storage Locations</em>, you can now delete the files associated with these old Storage Locations. Be sure to not delete the new Storage Location.</p>

<p>If you use MailStore’s internal backup feature go to the backup directory and delete all of the <em>Filegroup###</em> directories as MailStore’s backup will not do this automatically. The next time MailStore’s backup runs it will copy all files that are not in the backup directory.</p>

<h4 id="deleted-messages-or-folders">Deleted messages or folders</h4>

<p>If you deleted messages or folders (but not all Storage Locations) then you might consider running a Compact operation on your Storage Location(s).</p>

<p>You won’t get a lot of space back as much of the content would have been handled by the single-instance-storage system within the Storage Location but there will be some space recovered.</p>]]></content><author><name></name></author><category term="MailStoreServer" /><category term="MailStoreSPE" /><category term="MailStoreHome" /><category term="data management" /><summary type="html"><![CDATA[Introduction MailStore does not have any formal deduplication feature. While archiving MailStore checks each message in the source mailbox to see if it is in the archive in the correct location, skipping messages already in the archive, so in normal operation you should not see duplicated messages. However, if a user or a mailbox is renamed, you might end up with two different sets of folders for the same user, and although you can merge these together, if the two folders contain the same messages you’ll end up with duplicates. Because MailStore’s archiving process skips messages already in the archive we can export, delete and re-archive the messages, skipping the duplicated messages in most cases. You must first finish merging and organizing messages in MailStore before proceeding. Account You can perform these tasks as the admin, but you need to unblock access to the archive under Administrative Tools -&gt; Compliance -&gt; Compliance General. Alternatively an individual user can perform most of these steps. The user account must have Archive E-mail, Export E-mail, Delete permissions on their account and also Full Access to their archive. The admin will still need to be involved with certain steps, so it is best to perform the following steps as an administrator. Login to MailStore using whatever account you like, all of the steps will be performed in MailStore Client using the same account other than any Administrative Tools steps must be performed by an admin. Select the messages This step is optional, but in some cases you’ll be able to narrow down the range of messages using the search filter. If this is not possible, you can perform the following steps on individual folders, entire user archives, or across the entire MailStore archive as you prefer. Start by finding a few examples of duplicate messages, they should already be in the same folder and will have the same Date, but they may be archived on different dates, if so the goal is to determine the date that the duplication occurred. You can select a wider date range if you like. Go to Search E-mail Click the New Query button. Optional: Click the Folder […] button and select a starting point. You can leave this blank to work across the entire archive. Set the Date field to find messages older than the date the duplication occurred. Set the Archived Date field to find messages not older than than the date the duplication occurred. Click the Create Search Folder, save the search. You can review the search if you want, hopefully it will contain one instance of each duplicated message but it may contain both copies of the duplicated messages and/or it may contain messages that are not duplicated. This is fine. The goal is to make sure that at least one instance of every duplicated message is found. Stop MailStore Archive E-mail profiles from running It is highly recommend to stop all archiving tasks for the next steps. This can be done in more than one way: Launch the Configure the MailStore Server Service tool. Stop the service. Start in Safe Mode. This will stop all background tasks, and only allow admin users to login -OR- Go to the Archive E-mail panel Select Show Profiles of All Users at the bottom. Select each of the profiles that have a spinning circle, right click, select Manual. Whatever step you take, be sure to keep track of what you changed so you can revert the change later. Export the messages Go to Export E-mail Create a new E-mail Files export, Directory (File System), type EML files Set the Scope to the messages to be deduplicated, this can be the saved search folder created in the previous step or any other folder you like. Subfolders are included. Set the Target Folder to a local directory that has enough space. I’ll use C:\EML. The directory will be created as needed. Enable both Retain folder structure and Update existing export. Complete the profile and Run. A note about the folder you select: While most operations in MailStore Server use folders relative to the server, in this case we’re using a folder relative to the client. So for example, if MailStore Client is installed on your personal workstation and you export to C:\EML the files will be written to your workstation instead of the server. It is possible to use a mapped network drive, UNC path, or external drive, but I would recommend finding a machine with sufficient local internal storage instead as this process is dependent on your disk performance. Delete the messages Once the export finishes, return to MailStore Client and delete the messages. If you exported the entire MailStore archive Go to Administrative Tools -&gt; Storage -&gt; Storage Locations Right click on each Storage Location and Detach Create a new Storage Location. MailStore’s default name is in the format YYYY-MM, I would recommend staying with this format. Set it to Archive Here. If you used a saved search folder Navigate to the saved search. Select the top-most message (single click). Scroll to the bottom, Shift-Click the bottom-most message to highlight the range. Right click on any selected message and select Delete. If you exported any other folder Right click on the folder. Select Delete. Re-archive any missing messages Now we’ll re-archive the EML files that we exported. Go to Archive E-mail. Select E-mail Files. Select EML and MSG files. Select type MailStore Export. Set the folder (C:\EML in my case). Leave other options at their defaults (Include Subfolders, Read MailStore Headers enabled, Verify Signature disabled). Click Next &gt; Leave all Advanced Settings at their defaults. Click Next &gt; Under Target Archive select the admin user, or your own user. This archive is not used, but the field must be set. Complete the profile and Run. This will re-archive all of the exported messages, skipping messages already in the archive. If you were able to use a Search E-mail folder to select just a subset of messages then most messages will be skipped but there may be a few messages that were in the selected date ranges that did not get duplicated and we want to make sure we get these messages re-archived. It is possible that you won’t have any messages get archived here at all. When the profile finishes click the Details link, all messages should either be archived, or skipped because they were already in the archive. Cleanup Re-enable archiving You can now return MailStore to normal operation by undoing whatever steps you took in the Stop MailStore Archive E-mail profiles from running step. Free up disk space Depending on how you deleted the messages, you might be able to get some disk space back. Deleted all Storage Locations If you deleted all of your Storage Locations, you can now delete the files associated with these old Storage Locations. Be sure to not delete the new Storage Location. If you use MailStore’s internal backup feature go to the backup directory and delete all of the Filegroup### directories as MailStore’s backup will not do this automatically. The next time MailStore’s backup runs it will copy all files that are not in the backup directory. Deleted messages or folders If you deleted messages or folders (but not all Storage Locations) then you might consider running a Compact operation on your Storage Location(s). You won’t get a lot of space back as much of the content would have been handled by the single-instance-storage system within the Storage Location but there will be some space recovered.]]></summary></entry><entry><title type="html">Migrating a “live” MailStore Server using rclone</title><link href="/mailstoreserver/livemigration/rclone/" rel="alternate" type="text/html" title="Migrating a “live” MailStore Server using rclone" /><published>2023-01-31T00:00:00-07:00</published><updated>2023-01-31T00:00:00-07:00</updated><id>/mailstoreserver/livemigration/livemigration-rclone</id><content type="html" xml:base="/mailstoreserver/livemigration/rclone/"><![CDATA[<p>You cannot copy all of MailStore’s databases while the server is running, but if you’re comfortable with the command line, we can pre-stage the bulk of the database using rclone so that the required downtime is minimal.
<!--more-->
Be sure to first read the <a href="/mailstoreserver/livemigration/">Migrating a “live” MailStore Server</a> article first. Also available is <a href="/mailstoreserver/livemigration/robocopy/">robocopy instructions</a>.</p>

<h3 id="pre-staging-the-data">Pre-staging the data</h3>

<p>If you have any questions or have not used rclone before, take great care. In particular, understand that some commands (<code class="language-plaintext highlighter-rouge">sync</code>) not only copy but also delete files in the target.</p>

<p>First use <code class="language-plaintext highlighter-rouge">rclone config</code> to create a remote called <code class="language-plaintext highlighter-rouge">source</code> and <code class="language-plaintext highlighter-rouge">destination</code>, you can then perform the tasks from either machine, or a third machine entirely if desired.</p>

<p>One of the perks of rclone is that you can use it to copy and synchronize files across a variety of different protocols, the configuration and details are out of scope for this article.</p>

<p>You can use <code class="language-plaintext highlighter-rouge">--transfers</code> to the end of any of these commands to make them run multi-threaded, increasing the disk I/O and giving you delayed output.</p>

<p>I would also recommend <code class="language-plaintext highlighter-rouge">--progress</code> to monitor the progress in real time, and <code class="language-plaintext highlighter-rouge">-v</code> (verbose) to keep a record of the files copied.</p>

<p>You can also add <code class="language-plaintext highlighter-rouge">--dry-run</code> to see what will happen before you start.</p>

<p>We’ll start off by copying the content <em>.DAT</em> files, this will be a substantial portion of the database and most of these are not being modified (the most recent handful may be modified, and new ones are created, but that’s ok).</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rclone copy source:mailarchive destination:mailarchive --filter "- index*" --filter "+ *.dat" --filter "- **" --min-age 1d
</code></pre></div></div>

<p>You’ll possibly have a few errors, that’s fine and they can be ignored at this stage. Run it again if you want it to get caught up on any remaining files, but don’t worry about getting every last file.</p>

<p>Now we’ll run it again copying even more files, excluding the <code class="language-plaintext highlighter-rouge">.fdb</code> files as these will be modified. Additionally we’ll skip anything modified in the last day as these are likely to be modified again.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rclone copy source:mailarchive destination:mailarchive --filter "- *.fdb" --min-age 1d
</code></pre></div></div>

<p>Now we’re ready to stop archiving, so login to the existing server with MailStore Client, go to the <em>Storage Locations</em> panel, find the Storage Location marked as <em>Archive Here</em> and switch it to <em>Normal</em>, and copy everything except the <code class="language-plaintext highlighter-rouge">.fdb</code> files:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rclone copy source:mailarchive destination:mailarchive --filter "- *.fdb"
</code></pre></div></div>

<p>This will copy the remaining index files and recovery records even if they’ve been modified recently, but still skipping the .fdb files as they will be modified by MailStore again. You can run it again to retry any errors, but it isn’t critical.</p>

<h3 id="complete-the-transfer">Complete the transfer</h3>

<p>Finally, stop the service on the old server (and if you already installed the service on the new machine, make sure it is stopped there too).</p>

<p>This time we’ll use the <code class="language-plaintext highlighter-rouge">sync</code> command instead of <code class="language-plaintext highlighter-rouge">copy</code>, to ensure the destination matches completely.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rclone sync source:mailarchive destination:mailarchive
</code></pre></div></div>

<p>In this case there should be no errors, if you see any errors at all then make sure the service is stopped (on both sides) and run it again as needed until successful.</p>

<h3 id="leave-the-old-server-in-read-only-mode">Leave the old server in read-only mode</h3>

<p>At this point you can bring up MailStore on the existing server again and leave it serving users (but not archiving) by leaving the Storage Location in <em>normal</em> status. All archiving will remain paused.</p>

<h3 id="start-up-the-new-server">Start up the new server</h3>

<p>If you haven’t already, install MailStore Server on the new server, launch the <em>Configure the MailStore Server Service</em> tool on the new server, go to the security tab and initialize encryption, you’ll be prompted for the old encryption recovery key (by default, the product key although you may have a custom one configured). This does not transfer the license.</p>

<p>You can now start the service on the new server, run any pending database upgrades, compact, perform Storage Location consolidations, re-indexing, or whatever else is needed.</p>

<h3 id="complete-the-process">Complete the process</h3>

<p>Once you’re ready for the new server to start archiving, just switch the Storage Location that was formally set to <em>Archive Here</em> back to that status.</p>

<p>You’re now ready to uninstall the old MailStore Server installation, transfer the license and switch users over.</p>

<h2 id="licensing">Licensing</h2>

<p>From a licensing standpoint, only a single server can use the license at once, but you could acquire another license, or use a 30-day trial license on the new machine.</p>

<p>MailStore Server’s 30-day trials have one limitation that you won’t encounter when doing a normal 30-day trial: You cannot view messages that have been in the archive more than 30 days ago. You can still see the messages in the list, search, and access all administrative functions but the message bodies themselves have a trial overlay blocking access to the message body, which is fine for our purposes.</p>

<p>Hopefully everything here is clear, but if not, please don’t start the process until you really understand what it is going to do.</p>]]></content><author><name></name></author><category term="MailStoreServer" /><category term="data migration" /><summary type="html"><![CDATA[You cannot copy all of MailStore’s databases while the server is running, but if you’re comfortable with the command line, we can pre-stage the bulk of the database using rclone so that the required downtime is minimal.]]></summary></entry></feed>