Tuesday, February 18, 2014

How to solve the error 'ERROR: NMO Not Setuid-root (Unix-only)' on Oracle Enterprise Manager Cloud Control while contacting the EM Agent

I've just finished to setup at home an Oracle Real Application Clusters virtual environment formed by two nodes.

I would like to monitor and eventually execute some commands from another machine where I've installed the latest (at the moment of this post of course) available Enterprise Manager Cloud Console 12c that is Oracle Enterprise Manager Cloud Control 12c Release 3 Plug-in Update 1 (12.1.0.3) which contains the original 12c Release 3 (12.1.0.3) software binaries along with a few new plug-ins and updated plug-in versions.

I would like to compare the easy way to manage a Real Application Clusters from a GUI solution against the classic command line interface (CLI).

After I successfully deployed the agents from my Enterprise Manager Clod Control machine to the two target RAC nodes whenever I tried to perform some operations they resulted in the following error:
ERROR: NMO not setuid-root (Unix-only)

The following steps helped me to solve the problem.

On my nodes (demorac01 and demorac02) I created a local directory where EMCC deployed the agent. Let's have a look at the content of that directory in the second node:
[oracle@demorac02 agent12cr3]$ pwd
/u01/app/oracle/agent12cr3
[oracle@demorac02 agent12cr3]$ ll
total 28
-rw-rw-r--  1 oracle oinstall  178 Jun 22  2013 agentimage.properties
drwxr-xr-x  6 oracle oinstall 4096 Feb 15 10:36 agent_inst
drwxr-xr-x  3 oracle oinstall 4096 Feb 14 10:34 core
drwxr-xr-x 13 oracle oinstall 4096 Feb 14 23:04 plugins
-rwxr-xr-x  1 oracle oinstall  449 Feb 14 10:32 plugins.txt
-rw-r--r--  1 oracle oinstall  599 Feb 14 10:38 plugins.txt.status
drwxr-xr-x  5 oracle oinstall 4096 Feb 14 10:36 sbin
Inside the sbin directory there are other files that according to the Oracle documentations should be owned by root. During my local deploy they were set up to be owned by oracle user instead.
[oracle@demorac02 agent12cr3]$ cd sbin
[oracle@demorac02 sbin]$ ll
total 1372
drwxr-xr-x  3 oracle oinstall   4096 Feb 14 10:36 cfgtoollogs
drwxr-xr-x  3 oracle oinstall   4096 Jun 22  2013 install
drwxrwx--- 11 oracle oinstall   4096 Jun 22  2013 inventory
-rwxr-xr-x  1 oracle oinstall  22754 Jun 22  2013 nmb.0
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe.0
-rwxr-xr-x  1 oracle oinstall  57835 Jun 22  2013 nmhs.0
-rwxr-xr-x  1 oracle oinstall  40007 Jun 22  2013 nmo.0
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo.0
-rw-r-----  1 oracle oinstall     56 Jan 26 16:41 oraInst.loc
Before proceding with the solution adopted just note that the agent was running and collecting all the available info from the RAC node as you can see from the command below:
[oracle@demorac02 sbin]$ cd /u01/app/oracle/agent12cr3/core/12.1.0.3.0/bin
[oracle@demorac02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/agent12cr3/agent_inst
Agent Binaries    : /u01/app/oracle/agent12cr3/core/12.1.0.3.0
Agent Process ID  : 21139
Parent Process ID : 21096
Agent URL         : https://192.168.56.102:3872/emd/main/
Repository URL    : https://emcc.example.com:4900/empbs/upload
Started at        : 2014-02-14 10:39:20
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-02-15 10:30:47
Last attempted upload                        : 2014-02-15 10:30:47
Total Megabytes of XML files uploaded so far : 1.3
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 29.49%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-02-15 10:35:48
Last successful heartbeat to OMS             : 2014-02-15 10:35:48
Next scheduled heartbeat to OMS              : 2014-02-15 10:36:48

---------------------------------------------------------------
Agent is Running and Ready
To correct the permissions on the desired files you have to run again the file root.sh located under the software location of the Enterprise Manager Cloud Control.
[oracle@demorac02 sbin]$ su - 
Password: 
[root@demorac02 ~]# cd /u01/app/oracle/agent12cr3/core/12.1.0.3.0/
[root@demorac02 12.1.0.3.0]# ll
total 180
drwxr-xr-x  4 oracle oinstall 4096 Jun 22  2013 archives
drwxr-xr-x  2 oracle oinstall 4096 Feb 14 10:36 bin
drwxr-xr-x  6 oracle oinstall 4096 Feb 14 10:39 cfgtoollogs
drwxr-xr-x  6 oracle oinstall 4096 Jun 22  2013 common
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 diagnostics
-rw-r-----  1 oracle oinstall  430 Apr  7  2008 dummy.ic.linux.txt
-rw-r-----  1 oracle oinstall  412 Apr  7  2008 dummy.ic.txt
drwxrwxr-x  3 oracle oinstall 4096 Jun 22  2013 EMStage
drwxr-xr-x  6 oracle oinstall 4096 Feb 14 10:36 install
drwxr-xr-x  2 oracle oinstall 4096 Jun 22  2013 instantclient
drwxrwx--- 14 oracle oinstall 4096 Jun 22  2013 inventory
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 jdbc
drwxr-xr-x  6 oracle oinstall 4096 Feb 14 10:34 jdk
drwxr-xr-x  2 oracle oinstall 4096 Feb 14 10:34 jlib
drwxr-xr-x  2 oracle oinstall 4096 Jun 22  2013 js
drwxr-xr-x  2 oracle oinstall 4096 Jun 22  2013 jythonLib
drwxr-xr-x  5 oracle oinstall 4096 Jun 22  2013 ldap
drwxr-xr-x  4 oracle oinstall 4096 Jun 22  2013 lib
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 lib32
drwxr-xr-x  4 oracle oinstall 4096 Jun 22  2013 modules
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 network
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 ocm
drwxr-xr-x 10 oracle oinstall 4096 Feb 14 10:35 OPatch
drwxr-xr-x  7 oracle oinstall 4096 Jun 22  2013 opmn
-rw-r-----  1 oracle oinstall   56 Jan 26 16:41 oraInst.loc
drwxr-xr-x  7 oracle oinstall 4096 Feb 14 10:35 oui
drwxr-xr-x  5 oracle oinstall 4096 Feb 14 10:35 perl
drwxr-xr-x  2 oracle oinstall 4096 Jun 22  2013 prereqs
drwxr-xr-x 10 oracle oinstall 4096 Jun 22  2013 rda
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 relnotes
-rwxr-xr-x  1 oracle oinstall 1991 Feb 14 10:35 replacebins.sh
-rwxr-xr-x  1 oracle oinstall 1986 Jun 22  2013 replacebins.sh.ouibak
-rwxr-xr-x  1 oracle oinstall 1968 Jun 20  2012 replacebins.sh.template
-rw-r--r--  1 oracle oinstall 1968 Jun 20  2012 replacebins.sh.template.ouibak
-rwxr-xr-x  1 oracle oinstall 4079 Feb 14 10:35 root.sh
-rwxr-xr-x  1 oracle oinstall 3933 Jun 22  2013 root.sh.old
-rwxr-x---  1 oracle oinstall   10 Feb 14 10:34 root.sh.old.1
-rwxr-x---  1 oracle oinstall   10 Feb 14 10:35 root.sh.ouibak
-rwxr-xr-x  1 oracle oinstall 3874 Feb 14 10:35 root.sh.template
-rw-r--r--  1 oracle oinstall 3869 Jun 22  2013 root.sh.template.ouibak
drwxr-xr-x  5 oracle oinstall 4096 Jun 22  2013 stage
drwxr-xr-x  6 oracle oinstall 4096 Jun 22  2013 sysman
drwxr-xr-x  3 oracle oinstall 4096 Jun 22  2013 ucp
drwxr-xr-x  5 oracle oinstall 4096 Jun 22  2013 xdk
drwxr-xr-x  2 oracle oinstall 4096 Jun 22  2013 xsds
That file needs to be run as root user too.
[root@demorac02 12.1.0.3.0]# ./root.sh
Finished product-specific root actions.
/etc exist

Creating /etc/oragchomelist file...
Finished product-specific root actions.
Now if you go again under the previous sbin directory you can see that the owner is changed:
[root@demorac02 12.1.0.3.0]# cd /u01/app/oracle/agent12cr3/sbin/
[root@demorac02 sbin]# ll
total 1496
drwxr-xr-x  3 oracle oinstall   4096 Feb 14 10:36 cfgtoollogs
drwxr-xr-x  3 oracle oinstall   4096 Jun 22  2013 install
drwxrwx--- 11 oracle oinstall   4096 Jun 22  2013 inventory
-rws--x---  1 root   oinstall  22754 Feb 15 10:48 nmb
-rwxr-xr-x  1 oracle oinstall  22754 Jun 22  2013 nmb.0
-rwxr-xr-x  1 root   oinstall  43494 Feb 15 10:48 nmgsshe
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe.0
-rws--x---  1 root   oinstall  57835 Feb 15 10:48 nmhs
-rwxr-xr-x  1 oracle oinstall  57835 Jun 22  2013 nmhs.0
-rws--x---  1 root   oinstall  40007 Feb 15 10:48 nmo
-rwxr-xr-x  1 oracle oinstall  40007 Jun 22  2013 nmo.0
-rwxr-xr-x  1 root   oinstall 585242 Feb 15 10:48 nmosudo
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo.0
-rw-r-----  1 oracle oinstall     56 Jan 26 16:41 oraInst.loc
It's time to bounce the agent so the new settings could be used by the Enterprise Manager Clod Control during its operations. You must stop and start the agent as oracle user.
[oracle@demorac02 bin]$ id
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),492(vboxsf),54322(dba)
[oracle@demorac02 bin]$ pwd
/u01/app/oracle/agent12cr3/core/12.1.0.3.0/bin
[oracle@demorac02 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
[oracle@demorac02 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent ............................................................................... started.
[oracle@demorac02 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/agent12cr3/agent_inst
Agent Binaries    : /u01/app/oracle/agent12cr3/core/12.1.0.3.0
Agent Process ID  : 11011
Parent Process ID : 10961
Agent URL         : https://192.168.56.102:3872/emd/main/
Repository URL    : https://emcc.example.com:4900/empbs/upload
Started at        : 2014-02-15 10:51:36
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-02-15 10:53:23
Last attempted upload                        : 2014-02-15 10:53:23
Total Megabytes of XML files uploaded so far : 0.01
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 29.48%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-02-15 10:55:37
Last successful heartbeat to OMS             : 2014-02-15 10:55:37
Next scheduled heartbeat to OMS              : 2014-02-15 10:56:38

---------------------------------------------------------------
Agent is Running and Ready
I have a RAC with two nodes so the same steps need to be executed on the other node, demorac01 in my case. Here there are the wrong settings before running the root.sh script.
[oracle@demorac01 sbin]$ pwd
/u01/app/oracle/agent12cr3/sbin
[oracle@demorac01 sbin]$ ll
total 1372
drwxr-xr-x  3 oracle oinstall   4096 Feb 14 10:21 cfgtoollogs
drwxr-xr-x  3 oracle oinstall   4096 Jun 22  2013 install
drwxrwx--- 11 oracle oinstall   4096 Jun 22  2013 inventory
-rwxr-xr-x  1 oracle oinstall  22754 Jun 22  2013 nmb.0
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe.0
-rwxr-xr-x  1 oracle oinstall  57835 Jun 22  2013 nmhs.0
-rwxr-xr-x  1 oracle oinstall  40007 Jun 22  2013 nmo.0
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo.0
-rw-r-----  1 oracle oinstall     56 Jan 26 16:41 oraInst.loc
Run the root.sh script on demorac01 as root user.
[oracle@demorac01 sbin]$ su -
Password: 
[root@demorac01 ~]# cd /u01/app/oracle/agent12cr3/core/12.1.0.3.0/
[root@demorac01 12.1.0.3.0]# ./root.sh
Finished product-specific root actions.
/etc exist

Creating /etc/oragchomelist file...
Finished product-specific root actions.
Let's check the new settings.
[root@demorac01 12.1.0.3.0]# cd ../../sbin/
[root@demorac01 sbin]# ll
total 1496
drwxr-xr-x  3 oracle oinstall   4096 Feb 14 10:21 cfgtoollogs
drwxr-xr-x  3 oracle oinstall   4096 Jun 22  2013 install
drwxrwx--- 11 oracle oinstall   4096 Jun 22  2013 inventory
-rws--x---  1 root   oinstall  22754 Feb 15 10:54 nmb
-rwxr-xr-x  1 oracle oinstall  22754 Jun 22  2013 nmb.0
-rwxr-xr-x  1 root   oinstall  43494 Feb 15 10:54 nmgsshe
-rwxr-xr-x  1 oracle oinstall  43494 Jun 22  2013 nmgsshe.0
-rws--x---  1 root   oinstall  57835 Feb 15 10:54 nmhs
-rwxr-xr-x  1 oracle oinstall  57835 Jun 22  2013 nmhs.0
-rws--x---  1 root   oinstall  40007 Feb 15 10:54 nmo
-rwxr-xr-x  1 oracle oinstall  40007 Jun 22  2013 nmo.0
-rwxr-xr-x  1 root   oinstall 585242 Feb 15 10:54 nmosudo
-rwxr-xr-x  1 oracle oinstall 585242 Jun 22  2013 nmosudo.0
-rw-r-----  1 oracle oinstall     56 Jan 26 16:41 oraInst.loc
Bounce the agent on demorac01: stop and start it as oracle user.
[root@demorac01 sbin]# logout
[oracle@demorac01 sbin]$ cd ../core/12.1.0.3.0/bin/
[oracle@demorac01 bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Stopping agent ..... stopped.
[oracle@demorac01 bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent .......................................................... started.
Let's check the agent status.
[oracle@demorac01 bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3  
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/agent12cr3/agent_inst
Agent Binaries    : /u01/app/oracle/agent12cr3/core/12.1.0.3.0
Agent Process ID  : 13377
Parent Process ID : 13332
Agent URL         : https://192.168.56.101:3872/emd/main/
Repository URL    : https://emcc.example.com:4900/empbs/upload
Started at        : 2014-02-15 10:56:16
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-02-15 10:57:01
Last attempted upload                        : 2014-02-15 10:57:01
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 21.39%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-02-15 10:57:00
Last successful heartbeat to OMS             : 2014-02-15 10:57:00
Next scheduled heartbeat to OMS              : 2014-02-15 10:58:01

---------------------------------------------------------------
Agent is Running and Ready
So both agents are now running again on the RAC nodes. Let's verify that I solved my issue directly from the Enterprise Manager Cloud Control: the error "NMO not setuid-root (Unix-only)" was thrown the first time when I tried to start up a pluggable database.
 

Let's do the same step again. Here is the current situation: a CDB in open mode and the pluggable database PDB001 not open.
 

Click on the target name link named "DEMO.demorac_PDB001". It will open the following screenshot and information: "Database is not open. Regions that fetch real-time data from target database will not be displayed"

So I want to open that PDB. Point your mouse over "Oracle Database" menu, then "Control" and finally "Open/Close Pluggable Database".

Give the credentials to access the remote Oracle database.

It seems to work because the Enterprise Manager Cloud Console is currently showing a new page.
 

Because I want to open the pluggable database PDB001 on both the available instances I select both instances using the CTRL button, and then clicked on "Open" from the Action menu.
 

A new browser page will show the status of the operation in progress.

When the operation terminates successfully the "Open/Close Pluggable Database" page will show that the status of the pluggable database PDB001 is now OPEN.
 

Clicking on the target link of the pluggable database named "DEMO.demorac_PDB001" will show new activities are performed and so collected from the agent.
 

That's all.

13 comments:

Anonymous said...

Thanx a lot pal
tis is really helpful... :)

Lucien Jacquet said...

standalone database (11.2.0.3)
Cloud Control 12c (12.0.2.0)
weekly backup job managed by EMCC
job failed with same error

I followed your instructions as far as executing root.sh (I don't have a RAC environment), verified that the file ownership had changed, and re-ran the failed backup job. Success!

Thanks very much for the detailed, helpful resolution.

oakleyses said...

oakley sunglasses, prada handbags, oakley sunglasses, longchamp handbags, longchamp handbags, louboutin shoes, louis vuitton handbags, coach factory outlet, tiffany and co, coach purses, louis vuitton outlet, polo ralph lauren outlet, air max, prada outlet, longchamp outlet, oakley sunglasses cheap, ray ban sunglasses, louboutin outlet, michael kors outlet, michael kors outlet, tiffany and co, burberry outlet, christian louboutin shoes, coach outlet store online, jordan shoes, polo ralph lauren outlet, louboutin, kate spade handbags, michael kors outlet, coach outlet, air max, gucci outlet, michael kors outlet, ray ban sunglasses, chanel handbags, michael kors outlet, tory burch outlet, nike free, kate spade outlet, louis vuitton outlet, burberry outlet, louis vuitton outlet stores, louis vuitton, nike shoes, michael kors outlet

oakleyses said...

converse, air max, gucci, canada goose, juicy couture outlet, canada goose, wedding dresses, moncler, ralph lauren, lancel, montre homme, moncler, louboutin, oakley, karen millen, vans, coach outlet store online, air max, canada goose jackets, ugg, hollister clothing store, louis vuitton, baseball bats, hollister, rolex watches, juicy couture outlet, iphone 6 cases, canada goose uk, canada goose outlet, ugg, moncler, moncler outlet, timberland boots, hollister, supra shoes, moncler, canada goose, converse shoes, toms shoes, moncler, moncler, canada goose, ugg boots, ray ban, parajumpers, canada goose

Anonymous said...

Thanks for this post.

oakleyses said...

jordan shoes, christian louboutin, uggs outlet, michael kors outlet online, uggs on sale, louis vuitton outlet, louis vuitton outlet, louis vuitton, ray ban sunglasses, replica watches, christian louboutin uk, chanel handbags, michael kors outlet online, uggs outlet, longchamp outlet, nike air max, michael kors outlet, burberry handbags, tiffany and co, polo outlet, nike free, nike air max, ugg boots, oakley sunglasses, ray ban sunglasses, michael kors outlet online, oakley sunglasses, christian louboutin outlet, longchamp outlet, prada handbags, gucci handbags, prada outlet, oakley sunglasses wholesale, michael kors outlet, oakley sunglasses, kate spade outlet, christian louboutin shoes, louis vuitton outlet, tory burch outlet, ugg boots, michael kors outlet online, burberry outlet, cheap oakley sunglasses, louis vuitton, ray ban sunglasses, nike outlet, longchamp outlet

oakleyses said...

sac vanessa bruno, new balance, vans pas cher, ray ban uk, nike blazer pas cher, true religion outlet, michael kors outlet, true religion outlet, replica handbags, polo lacoste, oakley pas cher, coach purses, hollister uk, abercrombie and fitch uk, nike free uk, north face uk, louboutin pas cher, polo ralph lauren, hollister pas cher, nike air max uk, michael kors pas cher, nike air max, true religion jeans, timberland pas cher, nike air max uk, coach outlet, air max, michael kors, jordan pas cher, sac hermes, north face, lululemon canada, coach outlet store online, nike roshe, sac longchamp pas cher, nike air force, mulberry uk, hogan outlet, ralph lauren uk, longchamp pas cher, michael kors, converse pas cher, burberry pas cher, nike roshe run uk, true religion outlet, kate spade, nike free run, nike tn, ray ban pas cher, guess pas cher

Anonymous said...

Yup, this works.
My problem was that not all target where collected so I could not manage them in EM.
After correcting this all targets where available after some time.
Thanks!

Unknown said...

Best Way to Solve Oracle Database Startup Problem with Remote DBA Services
In the event that you are attempted to startup the Oracle database and got some specialized blunders at that point don't stress, simply attempt these beneath ventures to take care of your concern. To begin with you need to check every one of the organizers in the way as of now exist and furthermore check the consent on record way you are utilizing enable the database to peruse/keep in touch with it. All things considered, if as yet standing up to this issue at that point contact to Database Administration for Oracle or Online Oracle DB Support for fast and dependable help.
For More Info: https://cognegicsystems.com/
Contact Number: 1-800-450-8670
Email Address- info@cognegicsystems.com
Company’s Address- 507 Copper Square Drive Bethel Connecticut (USA) 06801

Anonymous said...

This can also occur if the set bits are on the mount. Look for nosuid, nosetuid, noatime, nodiratime, options and remove them. Bounce the box.

Quickbooks Phone Number said...

Nice & Informative Blog ! Are You facing issue in Quickbooks Phone Number ? We resolve all sorts of issues with effective solutions provided by our QuickBooks experts. Just give a call on +1-855-533-6333

jasonbob said...

golden goose
pandora
kd12
adidas yeezy
yeezy 350
canada goose jacket
yeezys
off white
yeezy shoes
nike kyrie 7

ESLAM UWK said...

أفضل أنواع خزانات المياه حيث يعاني الكثير من مشكلة انقطاع المياه أو عدم الوصول إلى المناطق النائية أو الطوابق المرتفعة ، لذلك قد يلجأون إلى شراء خزانات المياه التي تختلف في الشكل والنوع والحجم.