VMware vRealize Network Insight CVE-2023–20887
Introduction
I have recently identified and reported multiple vulnerabilities within VMware vRealize Network Insight by working with the Zero Day Initiative. Several of these vulnerabilities have been assigned a CVE:
In this post we’ll go over the exploitation process of VMware Aria Operations for Networks (Formerly vRealize Network Insight) specifically CVE-2023–20887, This is a chain of two issues which results in Remote Code Execution (RCE), Despite independently discovering and reporting the Pre-Authentication Remote Code Execution (CVE-2023–20887) vulnerability to the Zero Day Initiative (ZDI), along with several other vulnerabilities, I was outpaced by an anonymous researcher who reported it first. This post will examine the exploitation process of CVE-2023–20887 in VMware Aria Operations for Networks (formerly known as vRealize Network Insight). This vulnerability comprises a chain of two issues leading to Remote Code Execution (RCE) that can be exploited by unauthenticated attackers.
Vulnerability Analysis
The nginx configuration at /etc/nginx/sites-available/vnera restricts access to the /saasresttosaasservlet endpoint when called from :443.
The rule Only allows requests made from localhost.
Successful request to this endpoint will be proxy to the port 9090, an Apache Thrift RPC Server running on this port.
server {
[..SNIP..]
location /saasresttosaasservlet {
allow 127.0.0.1;
deny all;
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /saas {
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}The architecture for this RPC server mapping is as follow:
The RestToSaasCommunication responds when accessing the /resttosaasservlet on port 9090. This Thrift endpoint understands many procedures:
1 private static <I extends AsyncIface> Map<String, AsyncProcessFunction<I, ? extends TBase, ?>> getProcessMap(Map<String, AsyncProcessFunction<I, ? extends TBase, ?>> processMap) {
2 processMap.put("executeCommand", new executeCommand());
3 processMap.put("executeInfraCommand", new executeInfraCommand());
4 processMap.put("getDataSourceList", new getDataSourceList());
5 processMap.put("getDataSourceListWithWebProxyConfigured", new getDataSourceListWithWebProxyConfigured());
6 processMap.put("getDataSourceListByWebProxyId", new getDataSourceListByWebProxyId());
7 processMap.put("getDataSourceMapByDpIds", new getDataSourceMapByDpIds());
8 processMap.put("getAllDataSourcesMap", new getAllDataSourcesMap());
9 processMap.put("getOnDemandQueryResponseFromCollector", new getOnDemandQueryResponseFromCollector());
10 processMap.put("setDataSource", new setDataSource());
11 processMap.put("removeDataSource", new removeDataSource());
12 processMap.put("validateCredential", new validateCredential());
13 processMap.put("unpairPeer", new unpairPeer());
14 processMap.put("startDataSource", new startDataSource());
15 processMap.put("startDataSources", new startDataSources());
16 processMap.put("stopDataSource", new stopDataSource());
17 processMap.put("updateDataSource", new updateDataSource());
18 processMap.put("collectConfigNow", new collectConfigNow());
19 processMap.put("updateNode", new updateNode());
20 processMap.put("getNodesInfo", new getNodesInfo());
21 processMap.put("getCustomersNodesInfo", new getCustomersNodesInfo());
22 processMap.put("getProxyNodesInfo", new getProxyNodesInfo());
23 processMap.put("getFedPeerNodesInfo", new getFedPeerNodesInfo());
24 processMap.put("deleteNode", new deleteNode());
25 processMap.put("forcedDeleteNode", new forcedDeleteNode());
26 processMap.put("getDataSourceConfiguration", new getDataSourceConfiguration());
27 processMap.put("getDataSourceId", new getDataSourceId());
28 processMap.put("getDataSourceHostKeys", new getDataSourceHostKeys());
29 processMap.put("sendData", new sendData());
30 processMap.put("getTenantProxyDataSourceList", new getTenantProxyDataSourceList());
31 processMap.put("getSharedProxyDataSourceList", new getSharedProxyDataSourceList());
32 processMap.put("sendDataToGrid", new sendDataToGrid());
33 processMap.put("enableSupportTunnel", new enableSupportTunnel());
34 processMap.put("disableSupportTunnel", new disableSupportTunnel());
35 processMap.put("checkSupportTunnel", new checkSupportTunnel());
36 processMap.put("enableOnlineUpgrade", new enableOnlineUpgrade());
37 processMap.put("disableOnlineUpgrade", new disableOnlineUpgrade());
38 processMap.put("checkOnlineUpgrade", new checkOnlineUpgrade());
39 processMap.put("createSupportBundle", new createSupportBundle()); // urmum
40 processMap.put("sendUpgradeTargetManifest", new sendUpgradeTargetManifest());
41 processMap.put("getSystemInfo", new getSystemInfo());
42 processMap.put("createTenantSystem", new createTenantSystem());
43 processMap.put("deleteTenantSystem", new deleteTenantSystem());
44 processMap.put("createPlatformNode", new createPlatformNode());
45 processMap.put("sendNotifications", new sendNotifications());
46 processMap.put("setSystemPreference", new setSystemPreference());
47 processMap.put("toggleFipsMode", new toggleFipsMode());
48 return processMap;
49 }One of the available procedures is createSupportBundle , This procedure expects a structure which has been implemented below:
1
2 public static class createSupportBundle_args implements TBase<createSupportBundle_args, _Fields>, Serializable, Cloneable, Comparable<createSupportBundle_args> {
3 private static final TStruct STRUCT_DESC = new TStruct("createSupportBundle_args");
4 private static final TField CUSTOMER_ID_FIELD_DESC = new TField("customerId", (byte)11, (short)1);
5 private static final TField NODE_ID_FIELD_DESC = new TField("nodeId", (byte)11, (short)2);
6 private static final TField REQUEST_ID_FIELD_DESC = new TField("requestId", (byte)11, (short)3);
7 private static final TField EVICTION_REQUEST_IDS_FIELD_DESC = new TField("evictionRequestIds", (byte)15, (short)4);
8 private static final SchemeFactory STANDARD_SCHEME_FACTORY = new createSupportBundle_argsStandardSchemeFactory();
9 private static final SchemeFactory TUPLE_SCHEME_FACTORY = new createSupportBundle_argsTupleSchemeFactory();
10 @Nullable
11 public String customerId;
12 @Nullable
13 public String nodeId;
14 @Nullable
15 public String requestId;
16 @Nullable
17 public List<String> evictionRequestIds;
18 public static final Map<_Fields, FieldMetaData> metaDataMap;
19
20 public createSupportBundle_args() {
21 }
22
23 public createSupportBundle_args(String customerId, String nodeId, String requestId, List<String> evictionRequestIds) {
24 this();
25 this.customerId = customerId;
26 this.nodeId = nodeId;
27 this.requestId = requestId;
28 this.evictionRequestIds = evictionRequestIds;
29 }
30
31 public createSupportBundle_args(createSupportBundle_args other) {
32 if (other.isSetCustomerId()) {
33 this.customerId = other.customerId;
34 }
35
36 if (other.isSetNodeId()) {
37 this.nodeId = other.nodeId;
38 }
39
40 if (other.isSetRequestId()) {
41 this.requestId = other.requestId;
42 }
43
44 if (other.isSetEvictionRequestIds()) {
45 List<String> __this__evictionRequestIds = new ArrayList(other.evictionRequestIds);
46 this.evictionRequestIds = __this__evictionRequestIds;
47 }
48
49 }
50
51 public createSupportBundle_args deepCopy() {
52 return new createSupportBundle_args(this);
53 }
54
55 public void clear() {
56 this.customerId = null;
57 this.nodeId = null;
58 this.requestId = null;
59 this.evictionRequestIds = null;
60 }Which translates to the below structure:
struct {
customerId,
nodeId,
requestId,
evictionRequestIDs
}createSupportBundle as it’s name implies, will take care of support bundle creation:
1 public Result createSupportBundle(String customerId, String nodeId, String requestId, List<String> evictionRequestIds) {
2 ServiceThriftListener.logger.info("Request support bundle for customerId {} requestId {} nodeId {}", new Object[]{customerId, requestId, nodeId});
3 if (!evictionRequestIds.isEmpty()) {
4 for(int i = 0; i < evictionRequestIds.size(); ++i) {
5 if (!SupportRequestStore.isValidateRequestId((String)evictionRequestIds.get(i))) {
6 ServiceThriftListener.logger.error("Provided invalid evictionRequestId {}.", evictionRequestIds.get(i));
7 return new Result(ERROR_CODE.FAILED.getValue(), "Provided invalid eviction requestId " + (String)evictionRequestIds.get(i));
8 }
9 }
10 }
11
12 ServiceThriftListener.supportBundleExecutor.submit(() -> {
13 int cidInt = Integer.parseInt(customerId);
14 String nodeType = this.isLocalNodeId(nodeId) ? "platform" : "proxy";
15 SupportRequestStore.Policy policy = ServiceThriftListener.supportRequestStore.getPolicy(Type.SUPPORT_BUNDLE);
16 Integer maxFiles = policy != null ? policy.getMaxRequests() : null;
17 String vcfLogToken = this.getVCFLogToken();
18
19 try {
20 ScriptUtils.evictLocalSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);
21 ScriptUtils.evictPublishedSupportBundles(nodeType, nodeId, evictionRequestIds, maxFiles, vcfLogToken);
22 [..SNIP..]At line 21, the nodeId will be passed to the ScriptUtils.class#evictPublishedSupportBundles
The method does some basic checks to make sure the arguments are not empty and at lines 16, 18 constructs an interesting command.
1 public static synchronized void evictPublishedSupportBundles(String nodeType, String nodeId, List<String> evictionRequestIds, Integer maxFiles, String vcfLogToken) throws Exception {
2 Preconditions.checkArgument(NullOrEmpty.isFalse(nodeId, true));
3 Iterator var5 = CollectionUtils.emptyIfNull(evictionRequestIds).iterator();
4
5 while(var5.hasNext()) {
6 String r = (String)var5.next();
7 String filename = getSupportBundlePublishPath(getSupportBundleFilename(nodeType, nodeId, r, vcfLogToken));
8 Preconditions.checkArgument(!filename.contains("*"));
9 boolean deleted = ArkinFileUtils.delete(filename, FsType.DEFAULT);
10 if (!deleted) {
11 logger.error("Could not delete file {}", filename);
12 }
13 }
14
15 if (maxFiles != null) {
16 String evictCommand = String.format("sudo ls -tp %s/sb.%s.%s*.tar.gz | grep -v '/$' | tail -n +%d | xargs -I {} rm -- {}", "/ui-support-bundles", nodeType, nodeId, maxFiles);
17 if (CommonUtils.isPlatformCluster()) {
18 evictCommand = String.format("%s %s %s", "sudo /home/ubuntu/build-target/saasservice/cleansb.sh", nodeId, nodeType);
19 }
20
21 int evictRet = runCommand(evictCommand);
22 if (evictRet != 0) {
23 logger.error("Could not cleanup command {}, command returned {}", evictCommand, evictRet);
24 }
25 }
26
27 }The evictPublishedSupportBundles is vulnerable to a command injection by placing the nodeId inside a command at line 16 and 18, this command gets executed at line 21.
By crafting a Thrift RPC request it’s possible to exploit the createSupportBundle procedure, but as said before, access to this thrift endpoint from outside is restricted so I needed to get around this.
The Bypass
Normally in order to access the service and have it proxy using nginx, a request like this should be made:
https://VRNI-IP/saasresttosaasservlet --> MATCH location /saasresttosaasservlet ALLOW 127.0.0.1Which gets denied by the nginx rule, lets have a look at the nginx configuration one more time:
server {
[..SNIP..]
location /saasresttosaasservlet {
allow 127.0.0.1;
deny all;
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location /saas {
rewrite ^/saas(.*)$ /$1 break;
proxy_pass http://127.0.0.1:9090;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}If you haven’t noticed it yet, then look again, In order to bypass this rule, it’s possible to send a request like this:
https://VRNI-IP/saas./resttosaasservlet --> MATCH location /saas rewrite ^/saas(.*)$ /$1 PROXY_PASSThe nginx proxy will treat this as:
https://VRNI-IP/./resttosaasservlet