Illumio® Core
Version: 19.3.5
Release Notes
01/28/2021
Contents
Welcome ...5
What's New in This Release...5
Security Information...5
Product Version...5
Release Types and Numbering ... 6
Resolved Issues in Core 19.3.5 ... 6
Resolved Issues in Core 19.3.4...8
Resolved Issues in Core 19.3.4+H1-PCE... 8
PCE Resolved Issues in 19.3.4 ... 8
PCE Platform Resolved Issues... 8
Data Visualization Resolved Issues ... 9
PCE Web Console Resolved Issues ... 10
PCE Supercluster Resolved Issues in 19.3.4 ...10
VEN Resolved Issues in 19.3.4... 11
All Platforms: VEN Resolved Issues... 11
Solaris VEN Resolved Issues ... 11
Windows VEN Resolved Issues... 11
Linux VEN Resolved Issues... 12
AIX VEN Resolved Issues ... 12
Resolved Issues in Core 19.3.3 ... 12
Resolved Issues in Core 19.3.3+H1-PCE... 12
PCE Resolved Issues in 19.3.3 ... 13
VEN Resolved Issue in 19.3.3... 15
Resolved Issues in Core 19.3.2 ... 15
Resolved Issue in Core 19.3.2+H1-VEN ... 15
Resolved Issues in Core 19.3.2+H1-PCE...16
PCE Resolved Issues in 19.3.2 ...16
PCE Platform Resolved Issues...16
RBAC and Authorization Resolved Issues ...18
PCE Web Console UI Resolved Issues ...18
Containers Resolved Issues ...19
PCE Supercluster Resolved Issues in 19.3.2 ...19
VEN Resolved Issues in 19.3.2... 21
All Platforms: VEN Resolved Issues... 21
REST API Resolved Issues in 19.3.2... 21
Resolved Issues in Core 19.3.1 ... 22
Resolved Issue in Core 19.3.1+H5 ... 22
Resolved Issues in Core 19.3.1+H4... 22
PCE Resolved Issues in 19.3.1 ... 22
PCE Supercluster Resolved Issues in 19.3.1 ... 25
VEN Resolved Issues in 19.3.1... 26
Resolved Issues in Core 19.3.0... 27
Resolved Issue in Core 19.3.0+H7... 27
Resolved Issue in Core 19.3.0+H6 ... 27
Resolved Issues in Core 19.3.0+H5 ... 27
Resolved Issues in Core 19.3.0+H3 ... 27
Resolved Issue in Core 19.3.0+H2... 28
PCE Resolved Issues in 19.3.0 ... 28
PCE Supercluster Resolved Issues in 19.3.0 ... 33
VEN Resolved Issues in 19.3.0 ...34
Known Issues in Core 19.3.5... 37
PCE Known Issues ... 37
PCE Platform Known Issues ... 37
Data Visualization Known Issues ...38
Policy Known Issues ...38
PCE Web Console UI Known Issues ...39
Container Known Issues...44
REST API Known Issues ...46
VEN Known Issues...46
All Platforms ... 46
Linux VEN Known Issues ...47
Welcome
These release notes describe the resolved and known issues for the Illumio Core 19.3
releases, which include the Illumio Core 19.3.0, 19.3.1, 19.3.2, 19.3.3, 19.3.4, and 19.3.5 releases
and all their hotfixes.
Document ID: 14000-100-19.3.5
What's New in This Release
To learn what's new and changed in all the 19.3.
x
releases, see What's New in This Release
on the Illumio Technical Information portal.
Security Information
For information about known security issues, security advisories, and other security
guidance pertaining to this release, see the Illumio’s Knowledge Base in the Illumio Support
portal.
Product Version
Current PCE Version: 19.3.5 (LTS Release)
Current VEN Version: 19.3.5 (LTS Release)
Standard versus LTS Releases
19.3.5-PCE is a Long Term Support (LTS) release.
19.3.5-VEN is a Long Term Support (LTS) release.
The Illumio Core platform was previously known as the Illumio Adaptive Security
Platform (ASP). References to "Adaptive Security Platform" and ASP still appear in
these release notes.
• • • • • • •
For information on Illumio software support for Standard and LTS releases, see Versions
and Releases on the Illumio Support portal.
VEN and PCE Compatibility
Release Types and Numbering
Illumio ASP release numbering uses the following format: “a.b.c-d+e”
“a.b”: Standard or LTS release number, for example, “19.3” “.c”: Maintenance release number, for example “.1”
“-d”: Optional descriptor for pre-release versions, for example, “preview2” “+e”: Hot Fix release descriptor, for example, “+H1”, “+H2”, “+H3”.
Resolved Issues in Core 19.3.5
High CPU consumption issue resolved (E-74538)
Under some circumstances, the Windows Performance Monitor reported the VEN Platform Handler service's high levels of CPU consumption. Typically, this occurred when the FQDN cache manager performed FQDN cache maintenance. This issue has been resolved.
The disk was filled with core files, though this caused no disruption in PCE functionality (E-74367)
An internal monitoring process used to trigger a segfault due to a driver incompatibility. This resulted in excessive core files filling up disk space. However, it did not disrupt the PCE from functioning normally. This issue has been resolved.
Selecting Explorer from App Group failed when AppGroup mode was set to APP-ENV-LOC instead of APP-ENV (E-73815)
When you right-clicked and selected Explorer from Illumination, and the App-Group mode was set to APP-ENV instead of APP-ENV-LOC, you were not able to select the GO button. This issue has been resolved.
While the 19.3.0 and 19.3.1 PCEs are compatible with 19.3.4 VENs, they cannot be
used to distribute 19.3.4 VENs. If you are using these PCE versions in conjunction
with the 19.3.4 VEN, use a deployment mechanism other than the PCE repository.
• • • • • • • • • •
Some audit messages no longer appear in Windows 8 (E-73780)
In Windows 8 or later, the VEN will no longer enable audit for the following messages: MPSSVC Rule-Level Policy Change
IPsec Main Mode IPsec Quick Mode
An offline workload should have generated an event (E-73778)
A workload that was marked “offline” by the Decommission and IP Cleanup Timer, should have generated an event when the workload was actually removed from the policy. However, it did not. This issue has been resolved.
Browser memory issue when accessing large workload rules (E-73761)
Browsers ran out of memory when accessing large workload rules. This issue has been resolved.
Issue with unmanaged workloads in Explorer (E-73615)
Explorer sporadically failed to properly display IP address, hostname, or label information for some unmanaged workloads. This issue has been resolved.
An error message is displayed regarding an issue with ipt.rules.v6 (E-71048)
An error message on the VEN highlighted an issue with ipt.rules.v6 though no IPV6 addresses were configured. This issue has been resolved.
Object limit on workloads not enforced for 2x2 deployment (E-73482, E-66187)
The maximum number of workloads specified for a small 2x2 PCE cluster is known and included in Illumio documentation, but no hard or soft limits have been set to automatically enforce these limits. This issue was resolved in ASP 20.2.0 for customers running Illumio ASP in their datacenters.
The V-E score displayed as "syncing" instead of "N/A" (73481, E-65807)
In the Unmanaged Workload panel, the V-E score displayed as "syncing" instead of "N/A." This issue is resolved and a proper V-E score displays.
Events filtering result wasn't correct (E-72755)
Filtering events by the failure status could also return events that had a successful status. This issue is resolved. In this release, the PCE web console can correctly filters events to display only events that have the failure status.
Upgrade failed if the traffic datastore was a mount point ( E-72512 ) Upgrade would fail with a message like "Failed to configure role
citus_coordinator_slave_service". This occurred because the upgrade script needs to rename the traffic datastore directory to save the contents, and the directory cannot be renamed if it is a mount point. This issue is resolved. If the configured datastore is a mount point, it is automatically moved to a subdirectory under the mount point when the database services are configured the first time or during the upgrade.
Extra space in haproxy log (E-72476)
In the haproxy log, an extra space character was inserted between a POST request endpoint and the HTTP protocol, resulting in two spaces next to each other. This could break any
•
•
•
automated parsing of the log file. For example, "POST /api/v11/orgs/0/agents/activate HTTP/ 1.1" This is resolved. The extra space no longer appears.
Resolved Issues in Core 19.3.4
Resolved Issues in Core 19.3.4+H1-PCE
OpenSSL on host OS did not support EMS (E-73762) (Windows VEN version 18.2 or greater)
Some Windows workloads have current patches that require Extended Master Secret (EMS) support in order to reuse Transport Layer Security (TLS) sessions (see RFC 7627), but the versions of OpenSSL on supported PCE host operating systems, such as RHEL 6.10 and 7.6, do not support EMS. This caused API request processing to be delayed. The PCE web console UI slowed down, and the PCE could miss VEN updates when VENs did not wait long enough for the PCE to respond. The issue is resolved. haproxy has been upgraded to use OpenSSL 1.1.1, with support for EMS. EMS is not currently supported with FIPS-enabled PCE. In addition, a new field has been added in t he haproxy logs to show whether a request reuses an existing TLS session or creates a new one: tls-session-resumed or tls-new-session. For example:
192.0.2.52:39904 [01/Dec/2020:21:08:04.238] https~:9443 agent/
agent0/10.0.2.15:8080 5/0/9/666/680 204 267 - - ---- 1/1/0/0/0 0/0 {|keep-alive} "PUT /api/v1/agent_timed_work HTTP/1.1" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 tls-new-session
PCE Resolved Issues in 19.3.4
PCE Platform Resolved Issues
During HA, some required services might show up as optional (E-72308, E-61254) When a cluster was re-joined after being split, some required services would show as optional during an HA event. During the upgrade of the PCE software or HA recovery
procedure, some services including the ha-proxy may not be stopped completely. This might cause the services to fail to start subsequently when the PCE services are restarted. The corresponding processes had to be killed manually for the services to start successfully. This fix resolves the issue.
After upgrading the PCE, unable to bypass Illumio Login page when SSO configured for PCE (E-71202)
• • • • • •
to their corporate SSO login page. Before the upgrade to ASP 19.3.x, users could press enter or click Log In in the PCE web console Login page to be redirected to their corporate SSO login page. Users weren't required to enter any login credentials to be redirected to their SSO login page. This issue occurred for customers who configured third-party SAML-based
Identity providers (IdPs) to manage user authentication in their organizations. This issue is resolved. In this release, users can press enter or click Log In in the PCE web console Login page (without providing their usernames) to be redirected to their corporate SSO login page. NOTE: This fixed issue only applies to Illumio customers who install the PCE in their own datacenters. It does not apply to Illumio Secure Cloud customers, because they must always enter usernames in the Illumio PCE web console Login page regardless of whether they have SSO configured.
Upgrading the PCE and setting runlevel 2 could trigger service zombie (E-68957) When upgrading the PCE, the HAproxy process could fail to stop and trigger a service zombie (a process that is still running but the original file was deleted). This issue was very rare and only occurred when PCE files were deleted, such as during upgrade. This issue is resolved. The zombie process is deleted.
After changing PCE object limits, new values don't take immediate effect (E-68701) This issue is resolved. No user action is needed.
illumio-pce-ctl reset failed with DB nodes when using custom paths on a separate partition (E-65738)
During the illumio-pce-ctl reset operation, an error during file removal prevented the removal of other files. This issue is resolved. When encountering inaccessible files, the reset command will continue to clean up the rest of the data directory.
User-created event recorded as a system event (E-59208)
The "user.create" event recorded the "created_by" field as 'system' instead of 'user' and contained the corresponding information to identify the specific user who initiated the creation. This issue is resolved. The event is now recorded properly.
Restarting fluentd_data service unexpectedly triggered fluentd_source restart (E-57235) Restarting the fluentd_data service didn't work and triggered the fluentd_source service to restart. You had to use the kill and start commands instead. This issue is resolved.
Restarting the fluentd_data service no longer triggers the fluentd_source service to restart.
Data Visualization Resolved Issues
Rule coverage for ICMP and pair of IP List rules displayed incorrectly (E-72804, E-70973) In the draft view for Illumination and Explorer, the policy decision for the traffic flow between two workloads considers the case where a pair of IP list rules combine to allow the traffic. The pair of rules are written from the source workload to an IP list containing the destination and from an IP list containing the source to the destination workload. This calculation was
performed incorrectly for flows other than TCP, UDP, and All Services, for example, ICMP. This issue is resolved. The draft policy decision for other protocols has now been fixed.
•
•
•
App Group Map could take a long time to load when calculating stale data (E-72387, E-68479)
In previous releases, the PCE displayed a timestamp indicating when the data was last refreshed in the App Group Map page and a warning when the data was out-of-date. However, determining if the data was out-of-date could cause the App Group Map page to take a very long time to load when your managed environment had large numbers of workloads. Additionally, the PCE was simultaneously retrieving data about PCE health and that could cause a "PCE is busy" error to appear in the App Group Map page. These issues are resolved. In this release, the App Group Map no longer displays the "PCE is busy" error.
Additionally, the PCE loads the App Group Map page using a previously calculated map of the data. When choosing App Groups > App Group Map from the PCE web console menu, the App Group Map page no longer takes a long time to load when your managed environment has a large number of workloads.
PCE Web Console Resolved Issues
PCE web console Health showed "Replication Lag" as Unknown during sync (E-62368) When the PCE services are started on a database replica node, the "Database Replication Lag" section in the PCE web console Health page could incorrectly display the primary node and replica node information during the full base backup of the primary to the replica
database. The primary and replica node information could be inverted and one could be missing. This issue affected both the Policy database and the Traffic database. This issue is resolved. During the full base backup of the primary to the replica database, the primary node and replica node information is now displayed correctly and the replication lag is displayed as unknown.
PCE Supercluster Resolved Issues in 19.3.4
The initdb command did not run completely (E-69491)
The initdb command did not complete successfully and was not updated during subsequent attempts to start the database. The cause was a failure of the initdb operation and failure to clean up leftover files. On subsequent attempts, because of these files, the initdb operation was assumed to have already succeeded. This issue is resolved. If the initdb command fails, the leftover files are removed, so that subsequent attempts can start fresh.
• • • • • •
VEN Resolved Issues in 19.3.4
All Platforms: VEN Resolved Issues
VEN in enforced mode said its state was 'idle' (E-72916)
The PCE showed the workload in Enforced/Test/Build mode, but the command illumio-ven-ctl status showed the VEN was idle. This was caused by two VEN processes writing to the agent ID file at the same time. This issue is resolved. Processes that can write to the agent ID file are now managed to prevent unsynchronized writes.
Workload failed to upload support report (E-70967)
Generating a large support report from the PCE could take a long time, because the VEN's transport API timeout value was too short for large support reports. This issue is resolved. The timeout is now long enough.
UDP accepted traffic was reported with Visibility level blocked or off (E-70550)
Due to this issue, the VEN reported flows added by the firewall test, the flows that existed in the firewall state table but did not exist in the VTAP state table. This was usually triggered by a policy change. Illumination would report accepted UDP traffic when configuring the visibility level to blocked (flow_drops). This issue is resolved and no TCP/UDP accepted traffic
reported.
Solaris VEN Resolved Issues
/var partition could become 100% full due to pflog file (E-71666)
The /var partition could become 100% full due to the growth of the pflog0.pkt file. This was caused by unnecessarily starting the pflogd process, which writes to the pflog0.pkt file without any bounds. This issue is resolved, and we no longer start pflogd.
Windows VEN Resolved Issues
venPlatformHandler Crashed on Windows (E-72013)
venPlatformHandler could crash because of large numbers of SQL errors printed to platform.log. This issue is resolved. The excessive SQL errors are no longer logged. Windows VEN installation fails with CAQuietExec64: Error (E-71590)
Installing a Windows VEN could fail with the error CAQuietExec64. This occurred because an older version of the difxapi.dll had been installed on the machine where the VEN was to be installed. The VEN used API calls that were not supported by this older version of the DLL. This issue is resolved. The VEN now uses only API calls that are supported on all versions of difxapi.dll.
• • • • • • •
Linux VEN Resolved Issues
Coexistence with firewalld (E-69548, E-72019)
For Red Hat Enterprise Linux (RHEL) 7 or 8, a workload can run the firewalld or iptables services when its VEN is in Illuminated mode and the PCE it connects to has Firewall Coexistence mode enabled. Prior to this release, this feature was not supported. For more information, see the following topics:
About VEN Administration on Workloads in the VEN Administration Guide
to understand VEN modes
Firewall Coexistence in the PCE Administration Guide
AIX VEN Resolved Issues
Tampering alerts were seen for the VEN, complaining about hash mismatch (E-71607, E-71205)
On a completely empty policy (one without any peer workloads in Illumination or enforcement), the Illumio firewall management script would not be able to process the firewall and thus complain (incorrectly) about the hash mismatch. This issue is resolved. The Illumio firewall management script now correctly handles the case where there are no peer workloads in the VEN's policy and thus will not report tampering alerts.
Resolved Issues in Core 19.3.3
Resolved Issues in Core 19.3.3+H1-PCE
Long running database queries could get stuck in a PCE region (E-72752, E-72138)
In a Supercluster deployment, the way that the PCE handled notifications between member PCEs could result in database transactions not completing and degrade PCE performance. This issue is resolved.
Supercluster replication failure (E-72586)
In rare cases, a race condition could cause replication to fail between PCE regions in a Supercluster deployment. The failure could impact one or more member PCEs, causing the database service to crash and fail to restart. When this failure occurred, customers required Illumio Support to manually intervene and correct it. The underlying issue that caused the replication failure is resolved in this release.
Member and leader databases in Supercluster deployment could get out-of-sync (E-72494, E-72104)
This issue could cause a PCE in a region to apply the wrong policy for rules using virtual services to the workloads in that PCE region. This issue is resolved.
• • • • • •
PCE Resolved Issues in 19.3.3
node_available REST API requests weren't logged to the HAproxy log (E-70887)
In a production deployment, customers use the ASP REST API to run health checks from a load balancer; for example:
GET /api/v2/node_available HTTP/1.1
When the health check request was incorrect or incomplete, the request was reported via the HAproxy log. For example, the following incomplete request would be reported via HAproxy log:
GET /api/v2/node_available
This issue is resolved. In this release, the HAproxy log includes the following string: "GET /api/v2/node_available- HTTP/0.9"
Explorer UI/App Group Map was not reporting the right flow data (E-70191)
The Explorer returned up to 5 matching IP lists for flows by prioritizing the most specific IP lists. If the managed environment had more than 5 matching IP lists and the rule allowing the flow used, for example, the 6th IP List, the Draft Rules flow showed the traffic flow as blocked. To workaround the issue, you had to delete any unused lists. This issue is resolved by
increasing the number of IP lists returned to 50. In this release, the Draft view and Reported view show the flows and rule coverage correctly.
After PCE database failover, some services might fail to restart (E-69591)
After the PCE database failed over to a new data node, some services might not restart or become aware of the new PCE primary database. For example, in a Supercluster deployment, the agent-slony service on the data0 node might not restart after database failover. This issue is resolved. In this releases, the PCE ensures that a service is forced to check and update the location of its dependencies when the dependent services restart.
Duplicate services could be created in the PCE (E-69530)
When using Illumination to create a rule for a Windows service, a duplicate service could be created in the PCE. This situation only happened when a port-based service for the same port already existed in the PCE. This issue did not occur when using the Policy Generator to create rules for Windows services. The Policy Generate correctly detects existing services when creating rules. This issue is resolved. When using Illumination to create rules for Windows services, the PCE no longer creates duplicate services when the port-based service already exists.
Virtual Services weren't appearing in Illumination (E-69526)
When a virtual service didn't have a container workload bound to it, that virtual service didn't appear in the Illumination map. The Virtual Services list page (PCE web console > Policy Objects > Virtual Services) included the virtual service in the list. This issue is resolved. All virtual services correctly appear in Illumination, even those that are not bound to container workloads.
• • • • • •
In rare circumstances, the PCE could fail to fully recover after a power outage. The primary data node could get stuck in PARTIAL status and the other nodes could be in PARTIAL or STOPPED status. To workaround this issue, you had to restart the services on all nodes. This issue is resolved. In this release, the PCE fully recovers after a power outage.
During Supercluster leader join or restore, process didn't stop when detecting mismatched versions (E-69426)
When the versions didn't match, the process didn't stop and instead tried to join an older mismatched version. This issue is resolved. During leader join or restore, processes join the correct versions.
PCE allowed rules that included SecureConnect and User Groups (E-68874)
The PCE doesn't support enabling SecureConnect for user group-based rules (Adaptive User Segmentation); however, the PCE didn't prevent you from creating this type of rule. This issue is resolved. In this release, the PCE doesn't allow you to enable SecureConnect for user
group-based rules.
Deleting a VEN by using REST API /workloads/bulk_delete returned incorrect error (E-68693)
The bulk_delete workloads REST API is available for unmanaged workloads only. When you use this API to delete a VEN on a managed workload (a workload in the PCE that has an installed VEN), the response returned by the REST API was invalidly formatted JSON and included an incorrect error. This issue is resolved. When you inadvertently, use the /
workloads/bulk_delete API to delete managed workloads, the API returns the correct error that the action is not allowed.
Unable to specify “All” labels for the protected subnet for a SecureConnect Gateway (E-67219)
When adding or editing a protected subnet for the SecureConnect Gateway, you cannot specify “All” for a label type (for example, All Applications or All Environments). From the SecureConnect Gateway page, click Add in the Rules section. This issue is resolved. In this release, you can specify the All scope for labels when adding or editing the protected subnet. Explorer and Illumination Draft View were inaccurate for traffic between virtual services and IP lists/FQDNs (E-66913)
Explorer and the Draft View of Illumination could incorrectly indicate that traffic wasn't
allowed (draft rules didn't exist) when rules existed to allow the traffic. This issue occurred for traffic between specific instances of virtual services (not their labels or the App Groups they were in) and IP lists of IP addresses or FQDNs. When you selected a red line between a virtual service and an IP list of IP addresses or FQDNs in Illumination Draft View and added a rule to allow the traffic, the map still showed the traffic as not allowed. This issue is resolved. In this release, Explorer and the Draft View of Illumination correctly show when traffic is allowed between virtual services and IP lists/FQDNs.
The rpm -V command failed due to permission issues when run as ilo-pce (E-63230) Running the command rpm -V illumio-pce as the ilo-pce user failed to verify the package
• • • • • •
installation because the permissions were set incorrectly on some directories. This issue is resolved. In this release, rpm -V illumio-pce no longer fails when run as the ilo-pce user. When a locked user was unlocked, their API key was still unusable (E-60181)
This issue is resolved. When the REST API updates the user's locked status to unlocked, the PCE forcibly deletes the user's API key from memory so that the stale key is removed and the user gets a new key.
Stopping and restarting both core nodes resulted in PCE coming up as PARTIAL (E-59951) Stopping and restarting both core nodes caused the PCE to start up in the PARTIAL state. To work around the issue, you had to start the services on the PCE node data0. This issue is resolved. After stopping and restarting both PCE core nodes, the PCE returns to the RUNNING state.
Page filter wasn't retained on Labels, Label Groups, and Virtual Services List pages (E-59726)
The page filter wasn't retained on Labels, Label Groups, and Virtual Services list pages when adding, editing, or deleting objects and opening the corresponding details pages. This issue is resolved. When adding, editing, or deleting objects in the Labels, Label Groups, or Virtual Services list pages, and then opening their details pages, the PCE web console retains the page filters you specified.
When searching in Explorer or the Workloads page, the results didn't show up correctly (E-57995)
When the case in a search didn't match, search results in Explorer and the Workloads page didn't show up correctly. This issue is resolved. In this release, exact matches, regardless of case, always show up as the top search results in all pages.
VEN Resolved Issue in 19.3.3
Solaris VEN install: Dependency checks skipped when custom admin file used (E-61595) Illumio provides an admin file that is used as input when performing VEN installation. This file contains a flag, idepend=quit , that ensures dependencies are checked before installation. If a custom admin file was used instead of the one provided by Illumio, the idepend=quit flag might not be included, so dependencies might not be checked and important packages could be missing, causing the VEN to fail. This issue is resolved. This release fixed a dependency check that affected Solaris VEN installations under certain conditions.
Resolved Issues in Core 19.3.2
Resolved Issue in Core 19.3.2+H1-VEN
Windows VEN experienced a memory leak (E-70356)
• • • • • •
impacted the server; for example, causing the server to stop responding. This issue is resolved. VEN installation on Windows servers no longer causes memory issues.
Resolved Issues in Core 19.3.2+H1-PCE
PCE web console maps didn't display all labels of a workload (E-70114)
In the PCE web console, Explorer, the App Group Map, and Illumination could display only some labels of a workload. Consequently, traffic in the Draft view of a map could appear as blocked. That traffic was still allowed. This situation occurred after updating one or more labels for the workload; however, only some of the workload's labels were updated in the cache. This issue is resolved. In this release, the maps display all workload labels and indicate when rules exist to allow traffic.
Stopping and restarting both core nodes results in PCE coming up as PARTIAL (E-59951) Stopping and restarting both core nodes causes the PCE to start up in the PARTIAL state. Workaround: Start services on the PCE node data0.
PCE Resolved Issues in 19.3.2
PCE Platform Resolved Issues
Event exception appeared in the logs (E-67188) The following exception could appear in the logs:
Exception: ERROR: column "timestamp" is of type timestamp without time zone but expression is of type character varying
This exception could also cause PCE event pruning to fail. This issue occurred because the order of the timestamp column in the PCE database tables changed between releases. This issue is resolved. The timestamp column exception no longer appears in the logs or affects event pruning.
Database cache updates stop working when agent database is in recovery mode (E-67006) Posgtresql database connection errors caused redis cache updates to fail. This issue is resolved.
HA failures: Power off data-1 causes the other PCE nodes to get stuck in PARTIAL (E-66110, E-65837)
When you are upgrading the PCE, the recalc_replication process runs to promote newly added replication tables. Restarting slony services could mistakenly kill this process, due to a bug in the slony startup script. After database migration, when the upgraded PCE was brought to runlevel 2 or 5, slony services would not start properly. This failure would be reflected in the output from illumio-pce-ctl cluster-status. The slony startup script is fixed. Runlevel not set to 1 upon PCE start (E-65783)
• • • • • • • • •
of the product or after resetting all nodes of the PCE, the PCE would be stuck in status 'PARTIAL' with the runlevel shown as -1. This issue is resolved.
Installation stuck after database setup (E-65737)
Upon database setup, the following messages occurred: ERROR: cannot execute INSERT in a read-only transaction and `block in set_default_events_limits'. This issue is resolved. Severity for events generated for soft_limit.exceedednotification is INFO (E-65563) The severity ofsystem_task.prune_old_log_events event with
the notification soft_limit.exceededwas set to INFO instead of WARNING. This is fixed and the severity was changed from INFO to WARNING.
config_manager log file rotation (E-65317)
Some low-volume log files like config_manager.log were not being correctly rotated by the PCE Internal Syslog. This issue is resolved.
Potentially blocked traffic reported incorrectly in logs (E-65159, E-65320)
In logs, traffic events that should be showing "Potentially Blocked" were instead showing "Allowed". This issue is resolved.
PCE web console wouldn't start when FIPS mode enabled (E-65286)
On RHEL 7.x, when the host OS has FIPS mode enabled, the PCE fails to operate properly. Some services on core nodes fail to start, making the PCE and its Web UI
inoperative.The issue was caused by internal software conflicts when FIPS is enabled on RHEL 7.x systems. The issue is resolved by using a different hash function (SHA-512) that is
supported in FIPS mode on RHEL 7.x systems.
Policy sync on a Slave Node in a Cluster appears to not complete (E-64843)
Under some conditions including limited IOPS on data nodes, registration of some services may time out. This may cause some database services to be unable to sync (slave DB service in 'RUNNING' state) for a very long time. This fix resolves this issue.
Connections between PCE nodes in different datacenters (E-61731, E-62262)
When you have a PCE multi-node cluster with nodes deployed in different datacenters, TCP connections between the nodes could sometimes be intermittently disrupted. This issue is resolved.
Firewall endpoint emits an event for 503 in Listen Only mode (E-60774)
In Listen Only mode, when a VEN receives a lightning bolt or request firewall for any other reason (tampering, resync, startup, etc.), the VEN receives a 503 error code by design. This previously triggered a PCE event, even though no actual error had occurred. This issue is resolved. The PCE does not generate events for these expected error codes in Listen Only mode.
CentOS 7 security limits not applied (E-58337)
The expected session limits, configured in /etc/security/limits.conf, might not be in effect for the PCE. This could cause a severe performance impact that might go unnoticed for some time. See Session Limits Too Low in PCE Troubleshooting. This issue is resolved.
• • • • • • • • •
Data Visualization Resolved Issues
Traffic datastore mount prevents upgrade (E-65731)
If the traffic database data directory is mapped to a mount point, after upgrading PCE software from 18.2.5 to 19.3.0 or a later release, services failed to start on data nodes . This issue is resolved.
With Filtering, warning is provided before data is loaded (E-65416) This issue is resolved.
Error 500 when logging in to Illumination (E-63411, E-65429)
API calls made by the Illumination Web UI could fail under certain circumstances. This issue is resolved.
RBAC and Authorization Resolved Issues
SSO stopped working after upgrade (E-69684)
SAML SSO stopped working when upgrading the PCE to 19.3.2 or 19.3.3. Cause: Some customers had SAML certificates which did not contain newline characters. This worked fine with previous versions of Ruby, but caused issues in Ruby 2.7. This issue is resolved. We added a migration to reformat any invalid SAML certificates during the upgrade, to ensure SSO works smoothly after the upgrade.
Duplicate user name and external group name caused login to fail (E-66184)
Logging in to a PCE with a username that is identical to an external group name sometimes failed when Global read-only was disabled. This issue is resolved.
Apostrophe in group name causes crash in external groups (E-66177)
When there is an apostrophe (') in an External Group name, permissions could not be added to the group. This issue is resolved. You can now use apostrophes in group names.
PCE Web Console UI Resolved Issues
Virtual Server tab (E-68426)
In the Load Balancer Details page of the PCE web console, the Virtual Server tab is now working. When you click the Virtual Server tab, the tab loads as expected.
Icons swapped in scoped/global role removal confirmation popup (E-65786)
In the Web Console UI's Role-Based Access > Scoped Roles page, you can click Remove to delete a role. A confirmation popup is displayed with the scope, principals, and role. The Scope field in this popup incorrectly reversed the Location and Environment icons. This issue is resolved. The Location icon is shown next to the location name, and the Environment icon is shown next to the environment name.
Hard limit of 500 results for rule_search (E-65473)
• • • • • • •
NEN reported incorrectly on Health page in UI (E-65428)
When the NEN running on one data node goes down, the Health page reports the NEN running on a core node; however, the NEN is actually running on the other data node. This issue is resolved.
Wildcard in workloads filter (E-65232, E-65233, E-60827)
In the Workloads page of the Web Console, the asterisk (*) wildcard is supported in a filter expression for filtering the workload list; see Use a Wildcard to Filter Workloads . However, while the UI accepts the asterisk as a valid character, the filter always returned zero results. This issue is resolved. The wildcard now works as expected.
Events Configuration - Edit Repository dialog incorrect label (E-64994)
A dialog box for repository configuration had an incorrect text label, Edit instead of Save. This issue is resolved.
IP list “last modified on” date affects the rule search result when searching by IP (E-64834) This issue appeared if there were multiple IP lists with overlapping IP entries in the PCE. It could also appear if the entries in an IP list, when collapsed into the minimal CIDR
representation (e.g. 10.0.0.0, 10.0.0.1 --> 10.0.0.0/23), overlapped with the minimal CIDR representation of another IP list. If you wrote multiple rules using these different IP lists, the rule search result for an IP address within the overlapping range might non-deterministically return an incomplete set of rules. This issue is resolved.
Containers Resolved Issues
Issue with SecureConnect on Windows and UDP multicast (E-61686)
Some issues with using UDP encapsulation along with SecureConnect have been fixed. As a result, both Linux and Windows VENs will disable Force encapsulation . You must therefore allow ESP packet traffic in your security group. In addition, Windows will enable Require inbound and request outbound mode by default. You can disable this mode manually.
PCE Supercluster Resolved Issues in 19.3.2
Upgrading PCE Supercluster affected VEN pairing (E-67428)
After upgrading a PCE Supercluster deployment, newly installed VENs failed to pair with the PCE. The PCE returned an HTTP 500 error when VENs attempted to pair with it. Eventually, the VENs did successfully pair. This issue is resolved. Upgrading the PCE no longer causes VENs to initially fail to pair or the PCE to return an HTTP 500 error.
Can't generate support report from member PCE (E-65479)
In the Workload Summary page of the PCE web console, the Generate Support Report button for a given workload was grayed out on a member PCE. This issue is resolved. Users can now generate a support report on a member PCE even if the workload is paired to the member.
• • • • • • •
Changing user role on Supercluster member expectedly fails but appears to work (E-65438) As a Global Organization Owner, when you changed the role of another user on a
supercluster member, a 403 error was encountered as expected, and the user’s role was not changed. However, the UI displayed a Confirm Granted Access dialog, seeming to confirm the role change. This issue is resolved. The erroneous dialog is no longer displayed.
Upgrade can take a longer time to complete (E-55047, E-63356)
Due to the time it takes to replicate new database tables across all the PCEs, an upgrade could take longer than usual. The delay occurred when changing the PCE to runlevel 2 or 5 from runlevel 1 after upgrading the software. This issue is resolved.
Timeout during Supercluster restore (EYE-62815, E-65431)
When running the supercluster-restore command, a PCE could get stuck waiting for events to be confirmed on one or more PCEs. This could cause the Supercluster operation to time out. This issue is resolved.
PCE upgrade in Supercluster failed at database migration (E-65297)
When upgrading a Supercluster to 19.3.1, the database migration steps failed with the messages "No healthy db node" and "Failed to connect to database". This issue is resolved. You can now use the new --pce-fqdns-to-skip FQDN1,FQDN2... option to the
remap_supercluster_backup.rb script to give a list of fictitious FQDNs that you do not want in the remapped database.
Supercluster replication lag increases following database failover (E-65514, E-65430, E-62861)
When a data node was shut down, Supercluster replication lag could continue to increase. If the original node came back online and started database services during the failover, the database service could be running on both data nodes. This caused database issues and replication lag. This issue is resolved. Now, when a database node is promoted from slave to master, the slave node creates a lock so that the master node can not start while promotion is in progress. While evaluating promotion, if the slave node finds that the master node is back online, it cancels the promotion and remains as a slave node.
Listen Only mode not enabled automatically after Supercluster restore (E-63935)
The PCE did not automatically enable Listen Only mode after performing the supercluster-restore command. The issue was caused by conflicting internal signals about whether the PCE is standalone or a member of a Supercluster, since the PCE is both of these things at different stages of the restore operation. The issue could lead to the PCE delivering an incorrect policy. This issue is resolved.
Policy issues during GSLB regression (E-59624)
Fixed potential issue with serialization of concurrent updates from multiple regions to the same resource.
• • • • • •
VEN Resolved Issues in 19.3.2
All Platforms: VEN Resolved Issues
After VEN pairing, policy changes not applied (E-68080)
On SLES 12 VEN, the following error message is seen: Failed to apply policy changes. This issue is resolved.
VEN activation code logged in plaintext ( E-66004, E-66325, E-65911, E-65836) The activation code could be logged in plaintext. This occurred because when the PCE-generated pairing line and PCE repo are used to deploy VENs, the VEN activation code was passed in a query parameter. This issue is resolved. The authentication header is now used to avoid accidental logging of the activation code in plaintext.
Windows VEN requires deprecated root cert on workloads (E-65602)
The Windows VEN software previously used a Symantec time stamp server, which used a certificate that is now deprecated. The issue is resolved. The VEN now uses a digicert time stamp server.
Workload VEN took could remain in a state of active syncing (E-65296)
The VEN monitors workload interfaces and, when it detects a change that affects the workload firewall policy, the VEN reports the change to the PCE. In some cases, the VEN ended up reporting an interface change to the PCE even though the change did not affect the workload firewall policy. This situation created unnecessary load on the PCE. This issue is resolved. In this release, the VEN no longer reports unnecessary interfaces changes to the PCE; thereby reducing the load on the PCE.
NOTE: The VEN still reports to the PCE any interface changes that potentially affect the workload firewall policy.
Repeated warning messages in kernel logs (E-61079, E-65445) The following message was entered many times in kernel logs:
--physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
The issue has been resolved by fixing the firewall rules that were causing these kernel warnings.
REST API Resolved Issues in 19.3.2
If a locked user is unlocked, their API key is still not usable (E-60181, E-68973) This issue is resolved.
• • • • • •
Resolved Issues in Core 19.3.1
Resolved Issue in Core 19.3.1+H5
VEN upgrade failed resulting in VEN being uninstalled (E-65979)
This issue is resolved. A failed upgrade can no longer result in the VEN being uninstalled.
Resolved Issues in Core 19.3.1+H4
Upgrading PCE Supercluster to 19.3.1 affected VEN pairing (E-67428)
After upgrading a PCE Supercluster deployment to 19.3.1, newly installed VENs failed to pair with the PCE. The PCE returned an HTTP 500 error when VENs attempted to pair with it. Eventually, the VENs did successfully pair. This issue is resolved. Upgrading the PCE to 19.3.1 no longer causes VENs to initially fail to pair or the PCE to return an HTTP 500 error.
Event exception appeared in the logs (E-67188) The following exception could appear in the logs:
Exception: ERROR: column "timestamp" is of type timestamp without time zone but expression is of type character varying
This exception could also cause PCE event pruning to fail. This issue occurred because the order of the timestamp column in the PCE database tables changed between releases. This issue is resolved. The timestamp column exception no longer appears in the logs or affects event pruning.
PCE Resolved Issues in 19.3.1
Adding more than 25 labels to a label group required multiple steps (E-64818)
When creating a label group, you use the drop down list in the Add Member tab to select the labels you want to be part of the group. The list showed a maximum of 25 labels, even when more than 25 labels existed. This happened because the internal API that populates this list had a limit of 25 items. This issue is resolved. List that have more than 25 labels are correctly displayed in the PCE web console.
PCE does not provide feedback when a user labels a new workload (E-60188, E-64996) When a user is assigned the Workload Manager role with a defined scope, the PCE did not provide feedback when the user attempted to label a new workload outside their assigned scopes. When clicking Save, nothing happened and no warnings were provided. This issue has been resolved.
Manually modifying CSRF token in HTTP request resulted in 500 error (E-61217)
Manually modifying CSRF token resulted in a 500 Internal Server error when the PCE should have returned a 400 error. This was a usability issue only. This issue is now resolved.
• • • • • • • • •
PCE failed to start following upgrade to 19.3.x (E-61753)
The PCE failed to start following an upgrade to 19.3.x with the following error: ERROR: Consul migration failed. See consul_migration.log file for details.
You needed to re-run the illumio-pce-ctl start command on all nodes to workaround the issue. This issue is resolved.
Unable to clear traffic counter for an App Group in the SaaS Illumination map (E-63273) Clearing the traffic counter for an App Group in the Illumination map displayed a 502 Bad Gateway error message. Caused by sharding of data across multiple Redis instances in the Illumio Secure Cloud instance. There is a single Redis instance in on-premises installations, so the issue does not arise there. This issue is resolved.
Repairing a NEN didn't correctly set the token hash of the associated NFC (E-64949)
When updating or changing the NEN configuration, the associated Network Function Control (NFC) received the wrong token hash and was unable to authorize with the PCE. This issue is resolved.
Installing the PCE with custom paths was not supported by the PCE startup scripts (E-61929) The Illumio PCE RPM installation file and the configuration file runtime_env.yml can be placed in custom locations instead of the default locations. The --prefix option and the
ILLUMIO_RUNTIME_ENV environment variable are used to specify the custom locations. These options were being ignored in favor of hard-coded paths in the PCE initialization scripts. If the machine was restarted, both paths would revert to the hard-coded default values. This issue is resolved.
Syslog messages could have inconsistent timestamp formats (E-64284)
Some syslog messages did not use the GMT time zone format. This issue is resolved. All syslog messages now use the GMT time zone format.
PCE Health page didn't display information for master database (E-64568)
The Database Replication Lag section in the PCE Health page previously lacked node
replication information during the split-brain state of a migration. Symptoms would ultimately be resolved on their own afterwards. The root cause was a bug in the PCE UI implementation. This was addressed by updating the integration to reflect the correct information, preserving data integrity.
PCE didn't provide feedback when users tried to create workloads (E-64996)
When users who were part of the Workload Manager role tried to label workloads using labels they lacked permissions for, the PCE failed to display an error message. This issue is resolved. Policy Generator could display an incorrect date (E-64528)
When using the Policy Generator, the date in the Last Calculated field for an App Group incorrectly showed 12/31/1969 18:00:00. The issue persisted even after pressing the refresh button or trying different browsers. It was caused by a faulty condition check on the selected app group. This issue is resolved.
PCE core nodes did not ignore traffic_datastore option (E-62066)
The PCE core nodes correctly ignored traffic_datastore options in the runtime_env.yml file, except for data_dir: /<value>. The traffic_datastore: data_dir is intended for use only on data nodes, so its presence caused runtime_env.yml file validation to fail on core nodes. The
• • • • • • • •
issue is resolved by ignoring validation errors caused by traffic_datastore: data_dir on core nodes.
The traffic database for Explorer could end up missing data (E-63578)
Data pruning could operate incorrectly from a slave node and cause the wrong data to be removed. This issue is resolved.
Unmanaged workloads created by using Explorer did not use their hostnames (E-64491) Unmanaged workload names were created using the IP address from the traffic rather than the FQDN, but the FQDN was reported by the VEN in the traffic. This occurred because the process for creating unmanaged workloads looked for FQDNs resolved by the PCE, which did not exist for the traffic with FQDNs from the VEN. To fix the issue, unmanaged workload names are now prioritized as follows: 1. FQDN name from the VEN; 2. FQDN name resolved by the PCE; 3. IP Address.
Explorer search returned an error message (E-64537)
When searching for workloads (include or exclude) by specifying multiple CIDR blocks,
Explorer returned the error message "The filter property already exists in the filter. Try adding a different one." If the user clicked GO to proceed with the search, one of the CIDR blocks was removed before searching. This issue is resolved, and you can search using multiple CIDR blocks.
Fluentd service failed to write data (E-63383)
Although the limit for the buffer-directory wasn't reached, the Fluentd service failed to write data. This issue is resolved. Configuration parameters have been adjusted so the issue does not occur.
Opening the Explorer could display a 500 Internal Server error (E-63375)
Under certain circumstances, opening the Explorer (PCE web console left navigation > Explorer) could display a 500 Internal Server error. The 500 error also appeared in the PCE logs. The issue that caused the 500 error is resolved.
Name of field in Events data not descriptive enough (E-64708)
When soft limits or hard limits were exceeded for Events, the returned event data contained a field named num_found, which did not sufficiently describe the purpose or meaning of the field. To fix this issue, the num_found property of notification events of type soft_limit.exceeded an d hard_limit.exceeded is renamed to limit_value_found.
Could not remove all labels assigned to a Pairing Profile (E-61305, E-62275)
On creating a Pairing Profile with labels assigned to it or editing an existing Pairing Profile to add assigned labels, and then removing those assigned labels, on saving the changes the removed labels were still attached to the Pairing Profile instead of being removed. This issue is resolved.
Illumination showed traffic to only one virtual server (E-52834, E-61662)
When integrating F5 with Illumio ASP, importing multiple VIPs with different ports but the same VIP addresses caused problems. If two virtual servers had the same IP address but different ports, for example VIP1-80 and VIP1-443, Illumination would show both the virtual
• • • • • • •
servers, but all the traffic would be mapped to just one virtual server. The issue is resolved by using the IP address and port to uniquely identify a virtual server, not just the IP address. Incomplete policy push to VENs (E-64495, E-64807)
A policy might not include the complete list of peers to which flow was allowed, resulting in incorrectly blocked traffic. The issue was caused by a rare combination of factors: first, a race condition between a VEN stop and a VEN firewall request, and then the VEN being brought back online within the goodbye timeout window. The PCE is fixed to prevent these race conditions.
Traffic database constraint violation on insert (E-65091)
Flow data ingestion happened from both the data nodes and could possibly use the wrong temporary table. This was caused by a race condition causing constraint violation errors. This caused flow data loss. The issue is resolved.
PCE Supercluster Resolved Issues in 19.3.1
Pairing failed on Supercluster (E-60086)
Pairing and then unpairing VENs from a PCE in a Supercluster could fail. The unpaired VENs were vacuumed from the database after some time interval. If you then upgraded and tried pairing new VENs to the leader PCE, the pairing could fail. This issue was caused by a duplicate ID in the PCE database. This issue is resolved.
Supercluster replication lag reported incorrectly (E-59341)
If, for example, PCE 1 was lagging behind PCE 2, the situation was instead reported as PCE 2 lagging behind PCE 1. This issue is resolved.
Supercluster data restore command did not function correctly (E-62925)
Using the -skip-db-restore true and leader-file options with the
supercluster-data-restore command removed data from the member tables. This issue is resolved. The PCE now performs a validation check to prevent you from entering the leader-file option when the command also includes -skip-db-restore true.
Supercluster replication was failing (E-63299)
When stopping and restarting services on a node, some services did not de-register successfully and prevented the configuration of those services. This issue is resolved. Infinite loop during Supercluster join (E-60516)
While joining a PCE to a Supercluster, the operation got stuck and emitted the following message repeatedly: /var/tmp/illumio_pce_data/slonik_subscription_avenger_agent_prod: 84: begin transaction; - SSL SYSCALL error: EOF detected. This was caused by a failure to reconnect to the database during the Supercluster join. This issue is resolved.
• • • • • • • • • •
VEN Resolved Issues in 19.3.1
Erroneous IP list (E-62899)
Under certain edge conditions, the firewall optimizer could allow the creation of unused IP sets. This resulted in IP sets leaking into the IP sets created by the VEN. The IP sets were not named correctly, which caused policy application to fail. This issue is resolved.
(Solaris) Incorrect rule count in agentqualify.sh (E-55492)
When running the script /opt/illumio_ven/bin/.agent_qualify.sh, the output of the command showed an incorrect number in the row "number of rules found." This issue is resolved.
VEN processes couldn't receive signals when created by blocked signal processes (E-64716) VEN services started during installation could not receive signals and were not able to stop gracefully on RHEL 8. This issue is resolved.
Traffic was blocked for stateless rules that included All Services (E-63590)
The issue occurred when an All Services stateless rule was created and put into effect. Other traffic, such as TCP traffic, was not passed as stateless. This issue is resolved.
Custom rules could skip "Exclusive" rules (E-63742)
This issue affected VENs when the Firewall Coexistence feature was configured to use Exclusive mode. If you added custom rules to the firewall (using iptables native commands, such as -t nat -A PREROUTING -p tcp -j ACCEPT), they could be skipped due to the execution order of the rules. This issue is resolved.
(Solaris 11.4) VEN tampering checks failed for rules (E-62742)
A limitation in the Solaris awk command caused firewall tampering events to be triggered erroneously when large ipsets were needed in a policy. This issue is resolved.
The illumio-firewall script caused unnecessary error message on VEN startup (E-64263) A VEN startup script caused a spurious error message to be output during VEN startup at boot, with no effect on functionality of the VEN. The script was corrected to eliminate this error message.
The offline timer feature for the VEN could cause unexpected behavior (E-63870)
Setting the value for the VEN goodbye timer longer than the VEN disconnect timer could cause unexpected behavior; for example, after stopping services on a VEN, it incorrectly went offline after the disconnect timer duration passed. This issue is resolved.
GPO firewall was not detected (E-58498, E-62292)
In compatibility reports, the VEN couldn't detect the Firewall enabled by GPO when the name of GPO did not include "Firewall." This issue is resolved.
(AIX) VEN was not sending flow data to the PCE (E-62565, E-61699) The following flows were not sent to the PCE:
2019-10-18T11:16:04Z en0 I 0 4 10.14.250.127 10.15.16.42 6 10427 1531 0 T 800 U TBI=56 TBO=0
•
•
•
•
TBO=0
This issue is resolved.
Resolved Issues in Core 19.3.0
Resolved Issue in Core 19.3.0+H7
Delay occurred displaying Explorer results (E-63871)
When you upgraded from from 18.2.x to 19.3.0, the traffic database migration took longer than expected. This issue only occurred when you upgraded from 18.2.x to 19.3.0, but not for future upgrades. For example, upgrades from 19.3.0 to 19.3.1 would not encounter a similar delay. This issue is resolved. Upgrades from 18.2.x to 19.3.0 no longer experience an extended delay when migrating the traffic database.
Resolved Issue in Core 19.3.0+H6
Access across organizations due to same SAML group name (E-64191)
Users from two different organizations were able to access cross-organizational information if they had the same group name. This issue is resolved and users cannot access information across organizations.
Resolved Issues in Core 19.3.0+H5
NEN services do not start after restoring the PCE database node (E-63226)
After restoring a PCE database node, the PCE did not restore and replicate the auth_keys file, which prevented NEN services from starting. This issue is resolved. NEN services now
properly start after restoring a PCE database node. Explorer database was missing data (E-63693)
Under certain circumstances, the Explorer database could end up with missing data. This issue occurred when automatic data cleanup (also known as pruning) happened erroneously from the slave Explorer database. This issue is resolved. Explorer data no longer gets
incorrectly removed.
Resolved Issues in Core 19.3.0+H3
• • • • • • •
PCE RPM: 19.3.0+H3
PCE Web Console UI RPM: 19.3.0+UI1
The following issues have been resolved in 19.3.0+H3:
Fluentd service failed to write data (E-63383)
Even though the limit for the buffer-directory wasn't reached, the Fluentd service failed to write data. Configuration parameters have been adjusted so the problem does not occur. Opening the Explorer could display a 500 Internal Server error (E-63375)
Under certain circumstances, opening the Explorer (PCE web console left navigation > Explorer) could display a 500 Internal Server error. The 500 error also appeared in the PCE logs. The issue that caused the 500 error is resolved.
Viewing App Group in Illumination could display 502 Bad Gateway error (E-63382)
This issue only occurred when using the Illumio ASP Cloud (rather than running Illumio ASP in your datacenter). This issue is resolved. Viewing an Application Group in Illumio will no longer display a 502 error.
Resolved Issue in Core 19.3.0+H2
Illumination Map might be empty after upgrade (E-62454)
After upgrading from a previous version to a 19.3.0-PCE, App Group Map and Illumination might be empty. This was caused due to an index in Illumination not getting migrated properly. When new flows are posted, the flow is shown along with the old information. This issue is resolved in 19.3.0+H2 with proper migration.
PCE Resolved Issues in 19.3.0
In Explorer, couldn't filter with IP list that contains range (E-61274)
When using Explorer in the PCE web console, the error " Unexpected Input Validation Error" could occur when filtering with an IP list that contained range of IP addresses. This issue is resolved.
When using Explorer, exclude operation might be interpreted ambiguously (E-61197)
When writing a query in Explorer in the PCE web console that uses an exclude filter based on Labels, it was not clear when the query logic would be calculated using the OR rather than AND operator. This issue is addressed by displaying OR between the entities in the Explorer query exclusion boxes to make it clear when OR is being used.
Traffic was blocked on the Consumer side to the VIP (E-59917)
When enforcing an application that connected outbound to an F5 configured VIP rule, the traffic was blocked on the Consumer side to the VIP. This issue is resolved.
• • • • • • • • • • • •
"Application Label" became a label and prevented the selection of an actual label (E-59844) When selecting the Application Label to filter your workload selection on the Workloads page, the title "Application Label" became a label and prevented you from selecting an actual label. This situation usually occurred when you had multiple tabs open in the PCE web
console. Refreshing the page allowed you to select an application label. This issue is resolved. Database masking script could fail with NFS working directory (E-59621)
Masking script could fail when cleaning up temporary files, especially if the temp directory was an NFS mounted filesystem. This issue is resolved.
Frequent policy changes degraded performance (E-59618)
High policy churn could result in high disk usage and reduced PCE performance, caused by several factors, including degraded database vacuuming. Fixes have been made which have reduced the likelihood of adverse performance effects.
Traffic worker service restarted frequently (E-59614)
In the presence of a very high rate of database changes, the traffic worker service restarted frequently. This issue is resolved.
Syslog messages exceeding 65000 characters were not handled properly (E-58922) Previously, Syslog messages exceeding 65000 characters were not handled properly when using UDP as the transport protocol. This issue is resolved.
Error message remained on display even after missing ports were provided (E-58411) In the PCE web console, when configuring a Service, an error message was displayed when you left the Ports & Protocols field blank and clicked Save. This occurred because Ports & Protocols is a required field. After filling in Ports & Protocols, the error message did not
disappear, which was misleading. This issue is resolved. After entering the missing information for Ports & Protocols, the message disappears.
Policy check and rule coverage didn't include rules (E-58382)
When you added ANY-0/0 IP list in the Providers or Consumers fields of an existing rule that combined either Workloads or Labels, you could see the following issues:
In Illumination Draft View, the line turned from green to red indicating there was no rule defined even though the rule existed.
Policy Generator showed that the rule was not in the scope even though the rule existed.
Policy Check could not match the rule even though the rule existed.
This issue is resolved. When you add the Any 0.0.0.0/0 IP list to a rule, Illumination, Policy Generator, and Policy Check work as expected. The behavior above no longer occurs. Provision Status filter for Label Groups was not working (E-58308)
In the PCE web console, when viewing Label Groups and filtering on Provision Status, no label groups were shown, even though some Label Groups should have matched the specified filter. This issue is resolved. When viewing Label Groups and filtering on Provision Status, the matching Label Groups are displayed.
Rule Coverage was not displayed sometimes on the App Groups list page (E-58037)
When viewing a list of App Groups in the PCE web console, the Rule Coverage column might not have shown entries for some App Groups. When selecting coverage for an App Group,
• • • • • • • •
the progress spinner appeared, however, the coverage did not appear. This issue is resolved. The entries for App Groups correctly appear in the Rule Coverage column.
Extra events generated during session logout (E-58021)
Multiple events were generated for terminated user sessions due to both logout and timeout. At times, the events were duplicated or generated by system maintenance activity. This issue is resolved. Extra events are no longer generated during session logout.
Restoring db dump created on different PCE updates pce_scope (E-58015)
When you run the command illumio-pce-db-management restore --update-fqdn, you no longer need to manually update the pce_scope in the syslog_destinations table to reflect the new PCE FQDN. The new FQDN is now updated for you, and the new Local setting will correctly reflect the new FQDN.
Couldn't move virtual servers into or out of discovered group in Illumination (E-57975) When you tried to move a virtual server into or out of the discovered group, the virtual server remained in its original group. This condition was transient. The virtual service labels were correctly updated, and after several minutes, the Illumination map displayed the virtual service in the correct group. This issue is resolved. Virtual servers now move into and out of discovered groups without delay.
Clicking Service link in Virtual Service page displayed Resource not found (E-57942) After creating a Virtual Service and navigating to its details page, the Service link incorrectly displayed the “Resource not found. Start over” page. This issue is resolved. Clicking the Service link in a details page correctly displays the service's summary page even when the service is in draft state.
Active version Labels were not displayed for Virtual Services (E-57764)
After changing labels for a virtual service and clicking “View the active version,” the UI still displayed draft labels. This issue is resolved. Clicking the “View the active version” link now displays the active version of the labels.
Error message was vague for invalid label during VEN activation (E-57747)
When you specify an invalid label when activating a VEN with the command illumio-ven-ctl --activate, a more informative error message now appears in the Events UI on the PCE web console. The previous error message was "Workload could not be paired because activation data is invalid." The message now clearly says "Label not found."
Policy Commit error generated after SecureConnect Gateway Policy Restore (E-57674) Restoring a SecureConnect Gateway policy after deleting it resulted in a 406 Not Acceptable error when committing the restored policy. This issue is resolved. Restoring a SecureConnect Gateway policy succeeds when committing the restored policy.
Adding All | All | All | All with a Policy state All displayed unnecessary dialog box (E-57614) The “Add Firewall Coexistence and Policy State” dialog box appeared after specifying All Roles | All Applications | All Environments | All Locations with a Policy state of All. Because firewall coexistence does not require you to specify scopes, this dialog box was unnecessary. This issue is resolved. This unnecessary dialog box no longer appears.